At IO this year, I was particularly interested in App Engine (and Compute Engine) talks. On Friday, members of the App Engine team gave a content-rich talk about "Optimizing Your App Engine App". The talk leaned towards Python, but the speakers reassured the audience that most principles are the same for Java or Python.
They provided a lot of information, and this post isn't intended to transcribe everything, but I will mention take-away points that were meaningful to me. (The recorded presentation doesn't seem to be up on YouTube yet.)
The central performance trade-off is Latency vs. Cost. Generally speaking, if you use more resources to improve performance, you'll increase your costs. Also, though, improving efficiency can lower costs and can improve performance.
Datastore tips: minimize calls per request. Optimize frequently-used pages. Migrate to High Replication Datastore if you haven't already. Use key ids or names. Use run, not fetch. For one result, use get by keyname or ID. This last omits index reading.
Use projection queries, just return the fields you want. Performance benefits similar to access by keyname/ID. There are some limitations, including you can project only indexed properties, and its best for read-only, since you can't write it back to the datastore. Read the fine print in the docs.
Don't use offsets, use cursors. Don't index properties you'll never query.
Definitely consider NDB. This is a new one for me, I saw NDB as experimental and didn't look into it. NDB does a lot of caching behind the scenes for you, and makes coding for performance easier. NDB also supports embedding entities in a model. NDB has lots of async support.
One way to avoid queries altogether is to set cache control headers appropriately for data that won't change much.
Batch API requests. Use async workflows. NDB tasklets offer a lot of async support. Also, look at Brett Slatkin's 2010 fan-in talk.
Use the new pagespeed service to speedup serving static content.
Use the admin console performance settings to manage your app. Front-ends, idle instances, pending. DON'T COMBINE LOW-COST & LOW-LATENCY SETTINGS. Use one, set the other to auto.
For me, I'm going to spend some time studying NDB and then start to use it. I have a few use cases that will probably benefit from projection queries as well.
Thanks, App Engine team.
UPDATE: here is a list of App Engine related session videos, thanks to Dan Sanderson: