Google Data Feed Fetcher

Feed Title: Google App Engine Blog

Entry Title: Request performance in Java

Content: If you've been following the App Engine Java runtime group, you may have noticed some discussions about performance of the Java runtime. Many of you have complained about hard-to-predict DeadlineExceededExceptions, or unexpectedly slow requests that use a high amount of CPU. These issues often have the same root cause: App Engine is preparing a new instance of your code to respond an incoming request. We call this occurrence a "loading request". Since App Engine provides server resources on demand, there are several reasons why you might experience a loading request:

1. You just uploaded a new version of your application.
2. Your application may have gotten no traffic recently.
3. Your traffic has become high enough to need another JVM to scale.

<precompilation-enabled>true</precompilation-enabled>

Entry Title: App Engine SDK 1.2.8 Released Including New Admin Console features

Content: The App Engine team has been hard at work tackling our the issues on our tracker, tweaking APIs and closing bugs. In addition to a ton of bug fixes, 1.2.8 also includes:

Enhanced Admin Console - Users will notice new tools for managing tasks and queues created with the Task Queue API, and more visibility into index processing.

Improved Java Compatibility - This release adds support for new filter operators and inheritance to JPA and JDO as well as support for JAXB, the single most requested feature for the Java SDK.

This was also the first release we "previewed" with developers before formally rolling out changes. Thanks very much to all the developers that gave us feedback on the preview release. And if you're interested in helping with future prereleases, please subscribe to the discussion groups to see preview release announcements.

1.2.8 is now available for both Python and Java developers. Take a look at our release notes for details on all the changes included in this release.

Posted by Google App Engine Team

Entry Title: YouTube APIs + App Engine = YouTube Direct

Content:

When choosing a platform to power a new application that will be used by popular media sites, some of the criteria we look for are scalability, ease of deployment, and ease of management. As with the nature of news and media, traffic can go up and down without much warning and getting new content out of the door quickly is the name of the game. App Engine has greatly simplified development and deployment of YouTube Direct, a new tool built on YouTube's open APIs that allows media organizations to request, review, and re-broadcast user-submitted videos by embedding the upload functionality of YouTube directly into their own sites. By using the Google Plugin for Eclipse to "click and deploy" new code changes and App Engine's admin console to manage application data, our launch partners -- most of which had never used App Engine before -- were able to easily deploy their own instances of YouTube Direct.

To date, media organizations like The Washington Post, The San Francisco Chronicle, NPR, The Huffington Post and ABC's Good Morning America are using YouTube Direct powered by App Engine's Java runtime and we hope more organizations take advantage of the platform. Visit YouTube Direct's developer page to learn more about it. The project's source is also available on Google Code.

Entry Title: Community Update: deferred, open source, and more

Content:

Here are some of the recent developments from the greater developer community.

deferred.defer

Nick Johnson recently added a new module to the App Engine Python SDK which allows you to use the task queue service to execute deferred function calls. This library requires minimal configuration and makes it even easier to use tasks. Using it is as simple as calling deferred.defer with the function and arguments you want to call - for example:

from google.appengine.ext import deferred
deferred.defer(logging.info, "In a deferred task")

For more details, see the article.

Ruby as a runtime option

The JRuby App Engine project has done a lot of work making spin-up time less painful with the most recent release. They've also had some great contributions from the growing community, such as an Image API that is ImageScience compatible. Ruby programmers can now build applications using familiar tools like Sinatra and DataMapper, while having access to the full set of App Engine APIs for Java. Follow future developments on the jruby-appengine blog and watch for talks by the JRuby App Engine team at upcoming conferences: "Scaling on App Engine with Ruby and Duby" at RubyConf and "JRuby on Google App Engine" at JRubyConf.

An open source alternate environment: TyphoonAE

Open source efforts to provide alternate hosting environments for App Engine applications continue to expand, with the TyphoonAE project joining the existing AppScale project in providing a platform to run App Engine apps. TyphoonAE aims to provide a complete environment for hosting App Engine apps, and so far includes compatible datastore, memcache, task queue and XMPP implementations.

gae-query-pager

Juraj recently created and open sourced a library for the Java runtime which "generate[s] appropriate paging queries from a base query, and then using these queries to page through the data." This is intended to simplify iteration through all of the results in a very large result set.

AEJ Tools

If you love RESTful APIs then it would be worth your while to take a look at AEJ Tools. Their library provides "a server module which provides a Rest style access to your datastore data, and a client module which allows you to run queries remotely using the Groovy interactive console. It can be used to run queries, store new entities or as a data import/export tool."

An open source full text search library

The datastore related libraries just keep right on coming. Full text search is a highly requested feature for App Engine and those behind the open source appengine-search project have create a Python library which "[c]an defer indexing via Task Queue API. Uses Relation index strategy mentioned in Brett Slatkin's Google I/O talk."

Testing framework

Moving beyond the datastore, the developers at geewax have created an advanced testing framwork called GAE Testbed. It covers everything from isolated and quick unit tests to full scale end to end tests and builds on some of the great testing tools for App Engine which are already avilable.

Party Chat

In other news, the folks at techwalla have migrated their party chat app to App Engine and made the source code available. It uses the new XMPP API to simulate a chat room using your favorite XMPP client. You can read more specifics on their blog.

Other open source projects

Check out the open source projects wiki page for more open source App Engine applications and utilities. Since the last posting, nearly 15 new applications have been added, so keep checking in as new projects are added regularly. If you have a project of your own to add, just follow the instructions at the bottom of the same page.

That's brings our community update to a close. We're always interested in hearing about new and interesting projects which use App Engine. If you have something to share, drop us a line.

Entry Title: Audrey Roy Wins the App Engine + Twilio Developer Contest

Content:

Congratulations to Audrey Roy -- her "Price It By Phone" application won the Google App Engine + Twilio developer contest that ran from September 21st to October 4th. Audrey will receive $1000 in Google App Engine credit, and a Dell Netbook from Twilio.

Audrey hacked together her contest submission in under 48 hours after going to an art museum and discovering that the $60 box she wanted was only $37 on Amazon.com. She used the Amazon Product Advertising API to look up the best used and new prices for books by ISBN number, using any touch tone phone.

Look up Amazon.com book prices by calling (877) 265-8137. Enter the book's 10-digit ISBN number after the prompt. The voice will tell you Amazon.com's lowest new and used prices for the book. Then, to see the list of books that you've already price-checked, with links to their Amazon detail pages, enter your phone number at http://price-it.appspot.com.

You can find an interview of Audrey on Twilio's blog.

Thank you to everyone who participated in the contest.

Twilio runs weekly developer contests to encourage developers to explore the many use cases, technologies, and industries where voice applications can provide useful solutions. They announce a new category each Monday along with the previous week's winner. This week's category is "Twilio for Salesforce, with Appirio" and the deadline is midnight on November 19th.

Questions? Drop us a line at help@twilio.com

Entry Title: Task Queue Quota Increases

Content:

Since the Task Queue API launched in June, we've been thrilled with how many developers have started using the API. Many developers have asked for more quota so that they can better utilize the API within their apps. We're pleased that we can now address these requests by significantly increasing the Task Queue quotas and are happy to announce the following changes, effective from today:

• Free quota increased from 10K to 100K task insertions per day.
• For billing enabled apps, quota increased from 100K to 1M (one million) task insertions per day.
• Total execution rate (across all queues for a given app) increased from 10 to 20 task invocations per second.

Stay tuned for more enhancements to the Task Queue system in the coming months.

Entry Title: Find App Engine news and content at reddit

Content:

We're pleased to announce that you can now submit and comment on App Engine news and articles on a new App Engine specific section of the popular social media site, reddit. Our subreddit is for any articles, blog posts, or other content relevant to App Engine, and is actively moderated and maintained by the App Engine team. Registration takes seconds and allows you to submit, vote, and comment on new App Engine content. The top articles, as selected by the App Engine community, are shown in a widget on http://code.google.com/appengine/.

Be sure to check us out on the App Engine reddit - http://reddit.com/r/appengine

Entry Title: App Engine SDK 1.2.7 - Bugfix Release for Python

Content:

Due to two issues introduced in the 1.2.6 release of the Python SDK, we are releasing version 1.2.7 today. This is a bugfix-only release with just a few minor changes (no new features or functionality); all users of App Engine for Python should upgrade to the new version, available on our Downloads page.

The two issues were:

• The 1.2.6 release added a key argument to the Model class constructor that broke subclasses who were calling the constructor's private arguments with positional (not named) values. The fix is to restore the original ordering, and require that the new key argument be specified as a named argument only.
• The 1.2.6 release broke an interaction between remote_api and the local development app server.

Both are now fixed in 1.2.7.

There is no corresponding release for the Java language version of App Engine (the Java SDK remains at 1.2.6).

Entry Title: App Engine SDK 1.2.6 Released with Incoming Email, App Deletion, and more!

Content:

The App Engine team is psyched to present version 1.2.6 of our SDK for both runtimes, Python and Java. This releases contains plenty of improvements and bugfixes, along with some exciting new features:

Incoming Email - Your App Engine app has been able to send email for some time ... but now, with 1.2.6, your app can also receive email. After enabling mail as an inbound service (just like XMPP), users can email your application at whatever@yourappid.appspotmail.com. Inbound messages are converted to HTTP requests (again, just like XMPP) which you can receive via webhook handler. Docs for Python, Java.

Delete an App - Our developers have been quite vocal with their desire to delete an App Engine application once it is no longer used (It's one of the top 5 most requested features on our issue tracker). Well, this feature is now available via the Admin Console! Just visit the Application Settings page for more information. Please be careful when deleting an app - the appid can never be reused after deletion. Learn more.

Datastore Stats - You can now see more detailed statistics about how your application data is stored in the Admin Console. This information can also be accessed programmatically. Docs for Python, Java.

More details can be found in the Release Notes. To download the new SDK and get coding, please visit the Downloads page.

Entry Title: Interactive XMPP demos

Content:

We hope you've been enjoying the new XMPP API we shipped in App Engine release 1.2.5. We're always impressed with the cool scenarios our developers create!

A couple of Google engineers found the time to create cool App Engine demos using the XMPP API and we'd like to share them with you.

CrowdGuru - this app, courtesy of App Engine Developer Programs Engineer Nick Johnson, is a fun trivia game which crowdsources answers. Add crowdguru@appspot.com to your IM list and ask it a question. You can read up on full details of the design and implementation here.

Multi-Chat - David Symonds wasn't satisfied with simple person-to-person IM, so he built an IRC-like chat room system! Add multi-chat@appspot.com to your IM list and message it "/help" to find out how you can join channels and start chatting it up. Full source is available here, and a basic web interface is here.

Of course, please note that these apps are solely intended to demonstrate the power of the XMPP API - they are not official products in any form, nor are they supported. In particular, don't send any private or personally identifying information when chatting with these bots.

But definitely try them out and let us know what cool app experiences you are building with App Engine!

Entry Title: Build a Voice App in the Cloud and Enter the Twilio + App Engine Developer Contest

Content:

This week, Google App Engine and Twilio challenge you to build voice or voice-enhanced applications hosted on App Engine that use Twilio's voice application API to make and receive phone calls. Submit your app by October 4th to win a Dell Netbook from Twilio and $1000 of App Engine hosting credit from Google.

Twilio hosts a telephony in the cloud web service, allowing web developers to build scalable, reliable voice communications applications on web hosting platforms like Google App Engine. Put Twilio and App Engine together and you can build voice applications with features, scalability, and reliability that companies used to spend millions for.

What Are Voice Apps?

Voice apps can be as sophisticated as an airline flight status hotline or a company PBX, or as simple as getting the weather by phone. For inspiration, check out what previous contest winners have built:

• Advocate a Cause: Mobile Commons
• Get Answers on the Go: Ask-a-Human w/ Mechanical Turk
• Get in the Game: ExcitementLivesHere.com
• Have Some Fun: Phone Simon

Tips on Getting Started

• Get Started with App Engine for Java or Python
• Learn how Twilio works or read our Twilio Quickstart
• Check out an example Python/AppEngine Twilio application with source code: Weather by Phone
• Twilio has libraries for Python and Java to help you get started.

Contest Details

Where to Submit: http://contests.twilio.com/submit-your-twilio-project.html
Deadline: October 4th at MIDNIGHT (Pacific Time)
Prize: 1 Dell Netbook from Twilio and $1000 in Google App Engine hosting credit

Questions? Drop us a line at help@twilio.com

Guest Post by Danielle Morrill, Director of Marketing, Twilio

Entry Title: App Engine talks at a conference near you

Content:

It's September and the fall conference calendar is starting to fill up. Members of the App Engine team will present at these conferences: join us if you are in the area, and feel free to tweet @app_engine if you want us to participate in local developer community events around these dates!

Entry Title: Agile paddling with App Engine: lessons learned building the Canoe '09 website

Content:

I work for Norex, a web development company in Halifax, Nova Scotia. As part of our sponsorship of the 2009 ICF Canoe Sprint World Championships in Halifax (Canoe '09), we developed an application to deliver real time race results to standard and mobile web browsers. Thanks to the ability to rapidly develop and deploy a scalable application on Google App Engine, and to do so live during the event, what began as a small experiment became a huge success for Norex.

Our goal was to import instant results (directly from the timing system FinishLynx), upcoming race information, athlete bios, teams, and past race results from a system used by the race organizers, and to reformat that data for distribution to web browsers. We chose to develop in Google App Engine (with Django and appengine patch) and iUI (iPhone User Interface) frameworks. In fact, this was our first time deploying an app using any of these technologies! While we had a few weeks to build the prototype, we also had to deal with the added challenges of a varying data format and special cases that could occur during live races. As with many software projects, these specifications were not provided on a regular or consistent basis from the race organizers, so we had to anticipate the unknowns and work to adjust to the conditions at the time.

To give you a taste for just how agile we were able to be on App Engine, here are some events during the race which we were able to cope with:

• After the first day, the closed-circuit TV system which the race organizers planned to use for the event was down. Before it was back up and running, we were able to deploy a new view appropriate for display on the TVs.
• By the end of the first day, the application had handled 840,000 requests at a peak of 60 requests/second. This was already several orders of magnitude higher than we had expected. At the start of the second day, we reached 10% of our CPU quota before 7:45am and were processing 100 request/second. While we had originally cached only the main pages, with a few lines of code, we extended our caching strategy to cache every possible page request, a different version for each web platform, and re-deployed during the day's races.
• The rules of the events changed at 2pm on day 3 of the event to allow crew changes. By 4pm, I had reworked the models and views to support the change.
• At 11am on the final day of races, someone in the control booth manually deleted inaccurate records, but this created inconsistencies in the model causing errors. By 11:05am, we had deployed a patch to stop the errors, and by 1pm had deployed a version which gracefully failed when referenced objects were inadvertently deleted.
• Every night we deployed new views of the data as requested by race organizers, participants, and spectators.

We found that new deployments were possible (and quick) because we could stage versions of the application to the App Engine servers, and test these staged versions on live data. Switching from the live version to the staged version and back again takes seconds, thus we could paddle as confidently and rapidly as the racers.

The International Canoe Federation was so impressed by the dependability and versatility of the solution, that they elected to replace their usual channels with our application to serve up the official results for news organizations and to Sport Federation sites all over the world, and to be used internally for calculating medal counts.

All in all, we saw over 1,000,000 page views from 93 countries around the world, and experienced incredible stability and scalability from Google App Engine even when we were spiking 350 requests per second during the finals. It was a big win for Norex, and App Engine has proven itself to be a serious contender for developing scalable web applications. Thanks so much to the Google App Engine team for providing such an outstanding product!

Link to: ZAP Results Application

Guest Post by Adam Thurlow, Web Application Developer, Norex.ca

Entry Title: Migration to a Better Datastore

Content:

At Google, we've learned through experience to treat everything with healthy skepticism. We expect that servers, racks, shared GFS cells, and even entire datacenters will occasionally go down, sometimes with little or no warning. This has led us to try as hard as possible to design our products to run on multiple servers, multiple cells, and even multiple datacenters simultaneously, so that they keep running even if any one (or more) redundant underlying parts go down. We call this multihoming. It's a term that usually applies narrowly, to networking alone, but we use it much more broadly in our internal language.

Multihoming is straightforward for read-only products like web search, but it's more difficult for products that allow users to read and write data in real time, like GMail, Google Calendar, and App Engine. I've personally spent a while thinking about how multihoming applies to the App Engine datastore. I even gave a talk about it at this year's Google I/O.

While I've got you captive, I'll describe how multihoming currently works in App Engine, and how we're going to improve it with a release next week. I'll wrap things up with more detail about App Engine's maintenance schedule.

Bigtable replication and planned datacenter moves

When we launched App Engine, the datastore served each application's data out of one datacenter at a time. Data was replicated to other datacenters in the background, using Bigtable's built-in replication facility. For the most part, this was a big win. It gave us mature, robust, real time replication for all datastore data and metadata.

For example, if the datastore was serving data for some apps from datacenter A, and we needed to switch to serving their data from datacenter B, we simply flipped the datastore to read only mode, waited for Bigtable replication to flush any remaining writes from A to B, then flipped the switch back and started serving in read/write mode from B. This generally works well, but it depends on the Bigtable cells in both A and B to be healthy. Of course, we wouldn't want to move to B if it was unhealthy, but we definitely would if B was healthy but A wasn't.

Google continuously monitors the overall health of App Engine's underlying services, like GFS and Bigtable, in all of our datacenters. However, unexpected problems can crop up from time to time. When that happens, having backup options available is crucial.

You may remember the unplanned outage we had a few months ago. We published a detailed postmortem; in a nutshell, the shared GFS cell we use went down hard, which took us down as well, and it took a while to get the GFS cell back up. The GFS cell is just one example of the extent to which we use shared infrastructure at Google. It's one of our greatest strengths, in my opinion, but it has its drawbacks. One of the most noticeable drawback is loss of isolation. When a piece of shared infrastructure has problems or goes down, it affects everything that uses it.

In the example above, if the Bigtable cell in A is unhealthy, we're in trouble. Bigtable replication is fast, but it runs in the background, so it's usually at least a little behind, which is why we wait for that final flush before switching to B. If A is unhealthy, some of its data may be unavailable for extended periods of time. We can't get to it, so we can't flush it, we can't switch to B, and we're stuck in A until its Bigtable cell recovers enough to let us finish the flush. In extreme cases like this, we might not know how soon the data in A will become available. Rather than waiting indefinitely for A to recover, we'd like to have the option to cut our losses and serve out of B instead of A, even if it means a small, bounded amount of disruption to application data. Following our example, that extreme recovery scenario would go something like this:

We give up on flushing the most recent writes in A that haven't replicated to B, and switch to serving the data that is in B. Thankfully, there isn't much data in A that hasn't replicated to B, because replication is usually quite fast. It depends on the nature of the failure, but the window of unreplicated data usually only includes a small fraction of apps, and is often as small as a few thousand recent puts, deletes, and transaction commits, across all affected apps.

Naturally, when A comes back online, we can recover that unreplicated data, but if we've already started serving from B, we can't automatically copy it over from A, since there may have been conflicting writes in B to the same entities. If your app had unreplicated writes, we can at least provide you with a full dump of those writes from A, so that your data isn't lost forever. We can also provide you with tools to relatively easily apply those unreplicated writes to your current datastore serving out of B.

Unfortunately, Bigtable replication on its own isn't quite enough for us to implement the extreme recovery scenario above. We use Bigtable single-row transactions, which let us do read/modify/write operations on multiple columns in a row, to make our datastore writes transactional and consistent. Unfortunately, Bigtable replication operates at the column value level, not the row level. This means that after a Bigtable transaction in A that updates two columns, one of the new column values could be replicated to B but not the other.

If this happened, and we switched to B without flushing the other column value, the datastore would be internally inconsistent and difficult to recover to a consistent state without the data in A. In our July 2nd outage, it was partly this expectation of internal inconsistency that prevented us from switching to datacenter B when A became unhealthy.

Thankfully, there's a solution to our consistency problem: Megastore replication. Megastore is an internal library on top of Bigtable that supports declarative schemas, multi-row transactions, secondary indices, and recently, consistent replication across datacenters. The App Engine datastore uses Megastore liberally. We don't need all of its features - declarative schemas, for example - but we've been following the consistent replication feature closely during its development.

Megastore replication is similar to Bigtable replication in that it replicates data across multiple datacenters, but it replicates at the level of entire entity group transactions, not individual Bigtable column values. Furthermore, transactions on a given entity group are always replicated in order. This means that if Bigtable in datacenter A becomes unhealthy, and we must take the extreme option to switch to B before all of the data in A has flushed, B will be consistent and usable. Some writes may be stuck in A and unavailable in B, but B will always be a consistent recent snapshot of the data in A. Some scattered entity groups may be stale, ie they may not reflect the most recent updates, but we'd at least be able to start serving from B immediately, as opposed waiting for A to recover.

To Paxos or not to Paxos

Megastore replication was originally intended to replicate across multiple datacenters synchronously and atomically, using Paxos. Unfortunately, as I described in my Google I/O talk, the latency of Paxos across datacenters is simply too high for a low-level, developer facing storage system like the App Engine datastore.

Due to that, we've been working with the Megastore team on an alternative: asynchronous, background replication similar to Bigtable's. This system maintains the write latency our developers expect, since it doesn't replicate synchronously (with Paxos or otherwise), but it's still consistent and fast enough that we can switch datacenters at a moment's notice with a minimum of unreplicated data.

We've had a fully functional version of asynchronous Megastore replication for a while. We've been testing it heavily, working out the kinks, and stressing it to make sure it's robust as possible. We've also been using it in our internal version of App Engine for a couple months. I'm excited to announce that we'll be migrating the public App Engine datastore to use it in a couple weeks, on September 22nd.

This migration does require some datastore downtime. First, we'll switch the datastore to read only mode for a short period, probably around 20-30 minutes, while we do our normal data replication flush, and roll forward any transactions that have been committed but not fully applied. Then, since Megastore replication uses a new transaction log format, we need to take the entire datastore down while we drop and recreate our transaction log columns in Bigtable. We expect this to only take a few minutes. After that, we'll be back up and running on Megastore replication!

As described, Megastore replication will make App Engine much more resilient to hiccoughs and outages in individual datacenters and significantly reduce the likelihood of extended outages. It also opens the door to two new options which will give developers more control over how their data is read and written. First, we're exploring allowing reads from the non-primary datastore if the primary datastore is taking too long to respond, which could decrease the likelihood of timeouts on read operations. Second, we're exploring full Paxos for write operations on an opt-in basis, guaranteeing data is always synchronously replicated across datacenters, which would increase availability at the cost of additional write latency.

Both of these features are speculative right now, but we're looking forward to allowing developers to make the decisions that fit their applications best!

Planning for scheduled maintenance

Finally, a word about our maintenance schedule. App Engine's scheduled maintenance periods usually correspond to shifts in primary application serving between datacenters. Our maintenance periods usually last for about an hour, during which application serving is continuous, but access to the Datastore and memcache may be read-only or completely unavailable.

We've recently developed better visibility into when we expect to shift datacenters. This information isn't perfect, but we've heard from many developers that they'd like more advance notice from App Engine about when these maintenance periods will occur. Therefore, we're happy to announce below the preliminary maintenance schedule for the rest of 2009.

• Tuesday, September 22nd, 5:00 PM Pacific Time (migration to Megastore)
• Tuesday, November 3rd, 5:00 PM Pacific Time
• Tuesday, December 1st, 5:00 PM Pacific Time

We don't expect this information to change, but if it does, we'll notify you (via the App Engine Downtime Notify Google Group) as soon as possible. The App Engine team members are personally dedicated to keeping your applications serving without interruption, and we realize that weekday maintenance periods aren't ideal for many. However, we've selected the day of the week and time of day for maintenance to balance disruption to App Engine developers with availability of the full engineering teams of the services App Engine relies upon, like GFS and Bigtable. In the coming months, we expect features like Megastore replication to help reduce the length of our maintenance periods.

Posted by Ryan Barrett, App Engine Team

Entry Title: App Engine Launcher for Windows

Content:

As recently announced on the Google App Engine Blog, the 1.2.5 SDK for Python now includes a GUI for creating, running, and deploying App Engine applications when developing on Windows. We call this the Google App Engine Launcher.

About a year ago, a few of us recognized a need for a client tool to help with App Engine development. In our 20% time, a we wrote a launcher for the Mac. Of course, not all App Engine developers have Macs, so more work was needed. Thus, a new crew of 20%ers set off to write a launcher for our App Engine developers on Windows. Although Google is spread out across many offices around the world, it is surprisingly easy to connect with passionate engineers. For example, this new launcher for Windows has contributions from Dave Symonds in Australia, Mark Dalrymple on the east coast, and more engineers here in Mountain View.

The Windows launcher is written in Python and uses wxPython for its GUI. This means (with a little care) the launcher should work on Linux, and we'd like Linux developers to have the option of using it. Although we ship a binary of the Launcher for Windows (thanks to py2exe), shipping binaries for Linux is a bit more challenging. Fortunately, Google has a well-traveled path for solving this problem. For example, Google O3D provides binaries for Windows/Mac; it also provides source code and instructions for building on Linux. Thus inspired, we've open sourced the Windows launcher so that developers can use it on other platforms.

The goal of the launcher is to help make App Engine development quick and easy. There may be other tasks you'd like to integrate (e.g. run tests, re-encode images before deploying, etc) and with the launcher now open sourced, you can add them! We look forward to seeing contributions from the community.

We have also started the process of open sourcing the Mac version of the launcher. The source code is now available; however, it references some missing Google libraries, so it won't yet compile in its current state. Fortunately, those libraries have also been open sourced, so it will be possible to get things up and running using entirely open source code. I'll be using more of my 20% time to clean up the Mac launcher project in the coming weeks.

We hope the launcher will improve the workflow for App Engine developers. We also hope the source code will enable developers to adapt it to their needs, just as we do on Chrome, my main project. Finally, I am proud to continue a tradition of openness which began with my very first project at Google.

-- John Grabowski, Software Engineer

Let us know how the launcher works for you.

Open Source Code for the App Engine Launcher: for Windows and Linux, and for Mac OS X.

Entry Title: App Engine SDK 1.2.5 released for Python and Java, now with XMPP support

Content:

Today we are releasing version 1.2.5 of the App Engine SDK for both Python and Java, our first simultaneous release across both runtimes. We're excited about the great new functionality in this release ... including XMPP!

XMPP (or Jabber as it is sometimes known) is an open standard for communicating in real-time (instant messaging). One of the most popular API requests in the App Engine issue tracker has been support for XMPP, so today we are excited to mark that issue closed with the release of our new XMPP API for both Python and Java SDKs!

Like the other APIs that App Engine provides for developers, XMPP is built on the same powerful infrastructure that serves other Google products. In this case, we take advantage of the servers that run Google Talk. This new API allows your app to exchange messages with users on any XMPP-based network, including (but not limited to!) Google Talk. If you're currently participating in the Google Wave developer preview, you can also use the XMPP API to build bots that interact with your waves.

We've tried to make the XMPP API as simple as possible to incorporate into your existing Python or Java applications. We use the same webhook pattern that Cron and Task Queue already use: you send outgoing messages with an API call; you receive incoming messages as an HTTP POST. You can read more about the features the XMPP API in our documentation (Python, Java).

We're very proud of our first XMPP release, but there's still more work to do. In the future we hope to provide even more functionality to apps, such as user status (presence) and info on new subscriptions. If you have particular requests or feedback, please let us know.

Python developers have been processing tasks offline using App Engine Task Queues since mid-June, but until now the feature was not available in the App Engine for Java SDK. The 1.2.5 SDK now includes support for creating Tasks and Queues in our Java runtime.

If you're familiar with the Python Task Queue API, the Java version will look very familiar. We use the same webhooks pattern as with Cron (and now XMPP). The API provides a simple pattern for creating tasks, assigning them a payload and a worker, and inserting them into queues for scheduling and processing. There's lots of potential with the Task Queue API, so make sure to check out the Java Task Queue Documentation for more details.

With the 1.2.5 release, we are increasing the daily quota for Task Queue insertions to 100K for billing-enabled apps. Ultimately, we will raise the quota for both free and billing-enabled apps, but we hope this intermediate step opens up new scenarios for our developers using Task Queues.

Last but not least, we're very excited that 1.2.5 for Python now includes a Windows-based version of a useful tool that Mac OS X users have been enjoying for sometime: The Google App Engine Launcher!

This tool simplifies the process of creating new Python projects, testing them locally, and uploading them to the App Engine servers. In addition, we're releasing the source code for both Mac and Windows App Engine Launchers as open source projects. Watch this space for more details on where you can find the source, and how Linux developers can use the Launcher as well.

1.2.5 also includes the usual set of bug fixes, tweaks, and API polish. For a more detailed look at all the things that have changed this release, take a look at our release notes and, as always, let us know what you think!

Entry Title: New features in 1.2.4

Content:

When we released version 1.2.4 of the SDK earlier this month, a couple of new features were released that didn't quite make it into the release notes. We think they're really cool features, so we wanted to take the time to highlight them.

The first feature is that when logged into your app as an administrator, App Engine will include a couple of extra headers in all the HTTP responses it sends you. As an example, here's what I see in Firefox's Live HTTP headers plugin when I load my blog:
X-AppEngine-Resource-Usage: ms=293 cpu_ms=500 api_cpu_ms=236 X-AppEngine-Estimated-CPM-US-Dollars: $0.012320

The first header tells me that it took 293 milliseconds to generate the page, and 500 CPU milliseconds were consumed, of which 236 milliseconds were spent doing API calls - such as accessing memcache or the datastore. The second header tells me that App Engine estimates that serving 1000 requests like this one would cost about $0.01 if I was above my free quota - not bad!

You can view these headers using plugins such as Firefox's Live HTTP Headers or Firebug. Note that only logged in administrators see these figures - ordinary users, and users who aren't logged in, won't see them at all.

The second new feature is that we've enabled 'wildcard' domains for App Engine apps serving off appspot.com. What this means is that you can now create multiple subdomains for your App Engine app, and have them all served by the same application. Thus, your users can access 'myapp.appspot.com', or 'developer.myapp.appspot.com', or 'news.myapp.appspot.com', with your app deciding how to handle each request. No setup is required to use the wildcard domains - simply configure your app to serve up requests to these subdomains however you wish. You can detect which domain a user requested by looking at the 'Host' header in the incoming request - for example, in Python's webapp framework you can access this with self.request.headers['Host'].

Entry Title: Recent Happenings: TickTock, parallel queries, DryDrop, and more

Content:

Every month or so, we compile a list of interesting things in the community related to App Engine. Here are some recent projects and resources that you might find interesting:

Ubisoft's TickTock on Facebook

Ubisoft has released a fun game for Facebook called TickTock which challenges you to guess which of your friends have posted particular status updates. From the application page "TickTock is a Facebook trivia game that tests your knowledge of your friends. Use that knowledge to create playful devices to challenge others. Unlock items and create bigger challenges. Post them on your people's Facebook Walls, and see who knows their friends best!"

TickTock uses App Engine to serve the game content and store game information. We're excited to see this as a great example of a growing number of games which are using App Engine behind the scenes. Sign on to Facebook and give it a try.

Query the App Engine datastore in parallel with asynctools

Written by a group of engineers at VendAsta, asynctools is a rather nifty toolkit that allows you to execute datastore queries in parallel in the Python runtime. The interface is slightly more involved than using standard queries, but the ability to execute multiple queries in parallel can substantially reduce the render time for a page where you need to execute multiple independent queries.

Jason Collins has written a detailed article about how and why they wrote asynctools, which can be found here.

Easily host and update static sites on App Engine with GitHub

DryDrop is a new tool that lets you host your static site on Google App Engine and update it by pushing to GitHub. Thanks to GitHub post-receive hooks your App Engine site can be updated automatically when you push new content.

Installing Google App Engine applications into Google Apps

We have a new screencast up in the Developer's Channel on YouTube, this one covers Installing Google App Engine applications into Google Apps.

Recent open source developments

• TwitterFollow (source) is an article and accompanying sample application demonstrating Twitter's REST API in an App Engine app.
• Gaelyk (source), a "lightweight Groovy toolkit" that runs on App Engine's Java runtime, enables developers to deploy small Groovy applications on App Engine. It supports Groovy templates and Groovlets for separating content from presentation.

For more open source projects built on top of Google App Engine, see the Open Source Projects wiki page, and feel free to add your own open source projects to that page if you want to see them featured in an upcoming community update post.

Posted by Jeff Scudder, Nick Johnson, and Jason Cooper, App Engine Team

Java is a trademark or registered trademark of Sun Microsystems, Inc. in the United States and other countries.

Entry Title: Python SDK version 1.2.4 released.

Content:

We're psyched to release version 1.2.4 of the App Engine SDK for Python. Some highlights of what you'll find in this release:

• Remote API now supports more APIs (Images, Memcache, and URLFetch).
  • http://code.google.com/p/googleappengine/issues/detail?id=1596
• A local shell for easy access to the Remote API.
• Support for kindless queries (aka transaction descendant queries):
  • http://code.google.com/p/googleappengine/issues/detail?id=913
• PolyModel now supports keys_only queries:
  • http://code.google.com/p/googleappengine/issues/detail?id=1630
• Support for multiple inheritance for Model and PolyModel.
• Improved exception reporting in the bulkloader.
• Enhancement to SearchableModel allowing multiple properties to be indexed.
• Composite indexes are no longer required for some types of key queries.

For full details, please see the SdkReleaseNotes wiki page.

Downloads for Windows, Mac, and Linux are available on the Downloads page. This SDK update was for the Python runtime, so please post your feedback in the Python runtime discussion group.

We'd also like to draw your attention to two new articles from App Engine team members:

• Writing custom Property classes by Nick Johnson.
• Getting started load testing your application by Joe Gregorio.

Entry Title: Writing custom Property classes

Content:

The App Engine datastore API comes with a wide range of Property classes you can use to represent properties on your datastore models. Occasionally, though, you'd like to do something that the designers didn't think of. Fortunately, it's really easy to extend the Datastore API with custom Property classes. In this blog post, we'll demonstrate how to write your own Property class to store Python's decimal data type.

The Property interface is documented here, but there are two essential methods our DecimalProperty must override: get_value_for_datastore, and make_value_from_datastore. We also need to declare one field, data_type, that specifies the data type our property class will contain. Let's start with a straightforward implementation:

class DecimalProperty(db.Property):
  data_type = decimal.Decimal

  def get_value_for_datastore(self, model_instance):
    return str(super(DecimalProperty, self).get_value_for_datastore(model_instance))

  def make_value_from_datastore(self, value):
    return decimal.Decimal(value)

Note that in the case of get_value_for_datastore, we used super to call the parent class implementation, which handles the details of actually storing and retrieving the data stored in the model. We then simply convert the value to a string for storage in the datastore. In make_value_from_datastore, we reconstruct a Decimal object from the stored string.

That's all that's required for a basic property class! There is something missing from our example, though: We don't perform any validation to make sure that the user actually passed in a Decimal object. To do that, we override the validate method:

  def validate(self, value):
    value = super(DecimalProperty, self).validate(value)
    if value is None or isinstance(value, decimal.Decimal):
      return value
    elif isinstance(value, basestring):
      return decimal.Decimal(value)
    raise db.BadValueError("Property %s must be a Decimal or string." % self.name)

Again we start by calling the parent class's implementation, which performs some basic validity checks. Then we check if the value is a Decimal - in which case we return it - or a string, in which case we convert it to a decimal and return it. If the value is invalid, we raise a BadValueError.

Using our property class is as simple as using any other one:

class MyModel(db.Model):
  a_decimal = DecimalProperty()

model = MyModel()
model.a_decimal = decimal.Decimal("123.45")
model.put()

For a more in depth treatment of writing your own property class, see the article by Rafe Kaplan, Extending Model Properties.

Entry Title: Google App Engine for Java SDK 1.2.2 Released

Content: Greetings App Engine developers! We're pleased to announced the availability of a new Java SDK, available for download here. This is largely a bugfix release, but there are a few new features we think you'll be excited to get your hands on:

• The appcfg upload tool includes proxy support.
• JDO and JPA support an extension that lets you mark individual fields as "unindexed."
• At long last, the dev appserver has a data viewer. Start your app locally and point your browser to http://localhost:8080/_ah/admin to check it out.
• The local datastore now has the same transaction behavior as the production datastore (more info: http://code.google.com/p/googleappengine/issues/detail?id=1411)
• You can issue ancestor queries inside transactions.

Entry Title: A Day in the Cloud, new articles on scaling, and fresh open source projects for App Engine

Content:

The latest release of Python SDK 1.2.3, which introduced the Task Queue API and integrated support for Django 1.0, may have received a lot of attention, but there have been a number of other notable launches and events since our last update. Several of these are summarized below.

A Day in the Cloud

On June 24th, Google Apps and Virgin America invited people from around the world to participate in a one-day online scavenger hunt called Day in the Cloud. Competitors were each given one hour to solve various puzzles and find answers to a host of trivia questions, and prizes were awarded to the top five scorers. The site was hosted on App Engine's scalable infrastructure. For more information on the challenge, including photos and videos of participants competing mid-flight, check out the official Day in the Cloud Lounge.

Scaling series

App Engine enables you to deploy and run your Python- and Java-based web applications on Google's highly scalable infrastructure. There are a number of ways to optimize the performance of an application running on App Engine, some obvious and others more subtle. With this in mind, we recently released a series of articles promoting tips and tricks that you can use to make best use of resources like memcache and prevent potential issues such as datastore contention.

We also updated the articles listing since we recognized that the growing number of articles added since April 2008 (over 35 by last count) made it difficult to find content on a specific topic. The new listing allows you to filter the list of articles based on a particular tag or label, so you can easily find all articles related to the datastore, for example.

As always, we are interested in your comments, thoughts and suggestions for new articles, so please feel free to share.

GeoModel and geomodel-demo open sourced

Google Earth API support engineer Roman Nurrik recently open sourced his GeoModel project. GeoModel uses geohash-like objects called 'geocells' to provide a generalized solution for indexing and querying geospatial data in App Engine. GeoModel is optimized for the basic real estate finder/store locator use case, but can be adapted for use with large datasets.

Using GeoModel, developers can instantly geo-contextualize datastore models by simply inherting from the GeoModel class. Currently, entities can be associated with a single geographic point and subsequently indexed and filtered by either conformance to a bounding box or by proximity (nearest-n) to a search center point.

Other open source developments

• GaeVFS: an Apache Commons VFS plugin which implements a virtual file system on top of the App Engine datastore, using the datastore and memcache APIs.
• GraniteDS 2.0, now with support for App Engine for Java: Granite DS is an open source alternative to Adobe Lifecycle Data Services for Java EE application servers, and supports both server push and the persistence mechanism on App Engine for Java. For more information about the project and a link to a sample implementation, see the announcement post.
• jiql: a JDBC-based wrapper of App Engine's low-level datastore API which provides distributed database access via an interface familiar to many developers.
• pQg: a proof-of-concept PHP-based SQL to JDO wrapper plus sample application.

Posted by Jason Cooper, App Engine Team

Java is a trademark or registered trademark of Sun Microsystems, Inc. in the United States and other countries.

Entry Title: The new Task Queue API on Google App Engine

Content:

With release 1.2.3 of the Python SDK, we are psyched to present an exciting new feature - the Task Queue API. You can now perform offline processing on App Engine by scheduling bundles of work (tasks) for automatic execution in the background. You don't need to worry about managing threads or polling - just write the task processing code, queue up some input data, and App Engine handles the rest. If desired, you can even organize and control task execution by defining custom queues. A quick example:

   # for each user, add a task to send a custom email message
   for u in users:
       taskqueue.add(url='/work/sendmail',
          params=dict(to=u.email, subject='Hello ' + u.name, body='this is a message!'))
  
   return # finished now, emails will be sent offline when tasks execute

   ...

   # task handler at /work/sendmail, automatically called for each task created above
   class MailWorker(webapp.RequestHandler):
      def post(self):
         mail.send_mail(
            'from_me@example.com',
            self.request.get('to'),
            self.request.get('subject'),
            self.request.get('body'))

We're eager to help you learn and experiment with Task Queues. The team recently presented the feature at Google I/O and the video is now available (slides are here). We've also prepared a set of demos to help you get started. And of course, don't miss the feature documentation. The Task Queue API is Python-only for now; we'll have a Java language version available soon.

Please note that the Task Queue API is currently a Labs release - we want to get your feedback on its usability and functionality before finalizing the API. You'll notice that its Python import path currently includes the 'labs' module (google.appengine.api.labs.taskqueue). Before the feature is promoted out of Labs, we may need to:

• Change the quotas and limits which apply to Task execution (definitely, we hope to raise the number of Tasks you can use per day).
• Change the API itself if there are usability or functionality issues.
• Change how we bill for Task Queue usage.

Once we're ready to promote the feature out of Labs, we'll give weeks of notice and provide a transition path for our developers.

Last but not least, the 1.2.3 release is full of other new stuff as well! Stay tuned to the blog for more updates or check the release notes for exciting info on:

• Asynchronous urlfetch support
• Django 1.0 support

Visit the Downloads page to get SDK 1.2.3 now!

The Task Queue API is the first milestone of our plan to deliver rich support for offline processing. There's more to come, but we hope the simplicity and power of this first release opens a new range of possibilities for our developers. Try it out and let us know! We'll be watching the Group for your input.

Entry Title: App Engine @ Google I/O goodness for all to enjoy

Content:

Entry Title: Changing Quotas To Keep Most Apps Serving Free

Content:

Since App Engine launched, our goal has been to offer substantial free hosting resources to as many developers as possible. As previously announced, we are changing our free resource quota levels, effective on June 22nd. Our target level of free support has been 5 million page views per month for a reasonably efficient web application.

When we launched App Engine, we were intentionally generous in our free quotas, both because we didn't know resources usage of a typical request, and because we didn't offer a billing feature to allow developers to buy more resources for a higher-traffic app. Since our billing feature launched in February, developers with high-traffic applications can purchase additional resources far beyond our original fixed free quotas. Having been live for more than a year, we now have good empirical data on the average resource consumption per request, so we're able to set our quotas to more accurately support our 5 million page views target.

This change in the free quotas offered to every application is intended to allow us to continue to offer substantial free application hosting to any interested developer. We have grown a lot in the last year, with over 80,000 applications created, and with these changes to our free quotas, more than 90% of these applications will continue to serve completely free. To empirically determine reasonable levels for our quotas, we measured resource usage for all applications running on App Engine over a recent 7-day period. For each of the quotas, we took the highest daily average resource usage per HTTP request out of the 7-day period:

Multiplied by 5 million requests spread over a 30 day month, these per-request resource statistics translate to daily resource usage of 6.4 CPU-hours and 1.02 gigabytes of outbound data transfer. We top off the numbers by offering 6.5 CPU-hours and 1.07 gigabytes of outbound transfer. Though typically inbound data transfer is a small fraction of outbound data transfer, we made inbound and outbound data transfer symmetric to ease initial data uploads.

Finally — what do we mean by reasonably efficient applications? Simply put, efficient applications avoid unnecessary computation or data transfer, and two techniques common to efficient App Engine applications are the use of caching headers and memcache. Caching headers in an HTTP response prevent a user's browser from needlessly re-downloading information that hasn't changed, both speeding up the user experience and saving bandwidth. Similarly, memcache keeps frequently accessed data in a memory cache on App Engine servers, rather than always reading from disk in the Datastore, therefore saving CPU usage and Datastore load.

Again, these changes ensure we can keep our continuing promise to make it free to get started with App Engine.