1. Select the subset of the data you want (e.g. US academic) and download the XML file.
2. Fix some validation errors: take out the “<” and “>” within the text of nodes (I use TextMate for this).
3. Parse the XML file with a custom PHP script that creates a csv file with school, position, location and url to posting. Here is my simple script for the academic XML file.
4. Save the csv file produced by the script in step 3 as an Excel spreadsheet (Google Docs doesn’t like csv’s).  Add a “Latitude” and “Longitude” column to the spreadsheet.
5. Upload the Excel file to Google docs.
6. Follow these directions to populate the latitude and longitude of each position+location.
7. Publish the Google spreadsheet and save the unique id in the url that Google gives you.
8. Sign up for a Google Maps API account.
9. Follow these directions to produce a Google map of your postings.

UPDATE: This service may make this process a bit easier, produce cleaner maps and allow the incorporation of more information.

Maybe the AEA can follow these directions to produce these maps after this year.  Contact me with any suggestions or questions.

Finance: Supply of jobs falls, demand increases

Financial Services and Banking job postings have decreased 39% since March 2008.

Clicks on Financial Services and Banking jobs have increased 96% since March 2008.

Construction employment falls

builder Job Trends

builder jobs

Healthcare bucks the trends

healthcare Job Trends

healthcare jobs

Tax-related employment is volatile

tax Job Trends

tax jobs

SimplyHired.com also has a trends section with similar patterns.

Also see my own take on apartment rental supply and demand using online listings + web scraper.

The Amazon Mechanical Turk (MTurk) is one of the suite of Amazon Web Services, a crowdsourcing marketplace that enables computer programs to co-ordinate the use of human intelligence to perform tasks which computers are unable to do. Requesters, the human beings that write these programs, are able to pose tasks known as HITs (Human Intelligence Tasks), such as choosing the best among several photographs of a storefront, writing product descriptions, or identifying performers on music CDs. Workers (called Providers in Mechanical Turk’s Terms of Service) can then browse among existing tasks and complete them for a monetary payment set by the Requester. To place HITs, the requesting programs use an open Application Programming Interface, or the somewhat limited Mturk Requester site.

Why would economists find this service useful? An example from my own work might help. I am collecting all the individuals employed by new VC firms founded from 1992 – 2007. I need demographic information and employment histories for each VC partner. After scraping the web to get the VC firm websites and “team pages” I have a set of locations for an individual to find the online biography of each of some 3000 VC partners. I submit a job to Mechanical Turk that asks the Turk’er to go to the website, find the individual’s biography and copy and paste the text. Further jobs could ask the Turk to read the bio and answer questions like: 1) Does this person have an MBA or PhD? 2) Male or Female? 3) Founder of firm? Unfortunately, submitting HITs to the Turk system is somewhat difficult. Enter Smartsheet.

Smartsheet as a Frontend to Mechanical Turk

Although the Mechanical Turk service has a system for submitting HITs, it is a little cubersome and requires a bit of strange formatting steps. If you value your time just a little (say, $10/hour) I recommend using the project management webapp Smartsheet’s service SmartSourcing. Again, I refer to my research. I created an Excel file with columns like “Name”, “Firm Name”, “Website” and an empty column “Biography.” I upload this file to Smartsheet, select the rows for which I need biography filled in and walk through their SmartSourcing steps. In about 3 minutes I have submitted a HIT to 1000s of workers that will be complete in 12 hours. I can approve or reject the responses while I watch them populate the online spreadsheet.

Such a service is not free. For a $9.95/month fee (or $99/annual…ask them for a non-profit coupon), you get access to SmartSourcing. Then, on top of the standard Turk fees, you have Smartsheet charges:

Any paid Smartsheet subscriber has access to the Crowdsourcing feature. Monthly charges include Amazon fees plus the cost for work performed (number of tasks completed * cents paid per task) and a low Smartsheet processing fee ($.01 + 10% per task completed – usually $10-$30 per 1,000 tasks).

For example, I paid about $7 for 116 biographies of VC partners. It sounds relatively expensive, but this service has:

• increased the potential sample size of my studies
• expanded the set of possible control variables
• gives you the ability to request multiple workers per task for error checking
• kept me sane by outsourcing mundane data tasks

But the whole premise behind those Aaa ratings– that securitization could isolate a “safe” component of a pool of fundamentally risky loans– was deeply flawed. It is impossible to diversify away aggregate or systemic risk. All that the device did was to mislead investors into thinking they were protected from those nondiversifiable risks and push those risks onto the taxpayers and the Fed. Before we decide that securitization is the road out of our present difficulties, I would like a detailed and convincing explanation of why the past mistakes are not going to be repeated again.

And frankly, this plan doesn’t make survival look all that likely at anything remotely approaching GM’s current size–not if by “survival” we mean “weaning itself from taxpayer cash”.  The government can guarantee warrantees.  But it will be less effective at shedding all sorts of obligations than bankruptcy court.  The administration won’t be reorganizing the way a bankruptcy does; it will be negotiating.  To be sure, bankruptcy judges, too, negotiate, particularly among the various creditors.  But they do so with the power to cramdown firmly in their hands.  If the administration wants to wield that kind of power, it will have to find new and inventive threats to level at the creditors.  They may not find them.  And if they do, it’s not really in the best interests of the nation for the government to find lots of new and innovative ways to threaten private investors.

~ Megan McArdle on the  GM Bailout

I have been called the devil by strangers and “the Facilitator” by friends. It’s not uncommon for people, when I tell them what I used to do, to ask if I feel guilty. I do, somewhat, and it nags at me. When I put it out of mind, it inevitably resurfaces, like a shipwreck at low tide. It’s been eight years since I compiled a program, but the last one lived on, becoming the industry standard that seeded itself into every investment bank in the world.I wrote the software that turned mortgages into bonds. More here.

Newspaper Advertising Revenues Falling Across the Board

But how will Google ever make money? There’s the rub. The company’s adamant refusal to use banner or other graphical ads eliminates what is the most lucrative income stream for rival search engines. Although Google does have other revenue sources, such as licensing and text-based advertisements, the privately held company’s business remains limited compared with its competitors’.

Northern Light (www.northernlight.com), which matches Google in many search-engine competitions, sells archived articles from hundreds of publications and builds intranet portals for companies. AltaVista (www.altavista.com), another competitor, has integrated its search engine into a broader portal strategy. “There isn’t really good evidence, frankly, that companies focused purely on search, as Google has been, can support themselves with that model,” says Northern Light Chief Technology Officer Marc Krellenstein.

Business magazines not so good at forecasting: Article from  December 7, 2000

S&P 500 Inflation-Adjusted Earnings Down 80%