Building scalable PHP applications using Google’s App Engine - Notes

My Notes From: Building scalable PHP applications using Google’s App Engine
https://joind.in/10618

*Ian Barber
**Google Developer Relations

There's a command line tool for Google stuff.

Scale Up or **Down** based on usage.
Free to start
PaaS, IaaS

php-minisell.appspot.com

GeoLocation is included in the Request.
You don't get to see proper traffic because it isn't a proper server. It's an app.

Normal PHP gd/json/simplexml PHP 5.4.22
Read Only File System (Google Cloud Storage)

Google URL Fetching Service is automatically used for fetching.
Can't use cURL

Modules - His "Single Favorite Thing" about AppEngine
Scaling modules manually (fixed) or basic (max num) allow control

YAML configs.
Handlers are like Mod Rewrite - Redirect everything to index.php
Static Content you need access to, use static_dir config

app yaml
api yaml
dispatch yaml
queue yaml
cron yaml
dos yaml (DDOS)

**Storage
CloudSQL (MySQL 5.5/5.6)
Cloud Data Store (Schemaless, NoSQL) - FREE TIER!
CDS available via REST API

Reading -
Google_Client() is a bit overly verbose (support for PHP 5.2)
Authentication you may as well use Google_Client for that because it's a huge pain.
Data Store using the Google_Client means highly optimized connection.

Caching -
Memcache - One big pool. No charge for usage. You can pay for fewer evictions by paying for storage.
'Cache-Control' http header. (Classical Gas)

Writing -
Google_Service_Datastore_CommitRequest()
Faster if you type your properties (int/string/etc)
Mutation Upsert

Logging -
Developers Console already takes care of it.
syslog writes to the dev console logging

Polymer Ajax -> Just another Buzzword JS/Display Framework?

Sign In -
Oath - PHP Google Service

Deployment -
appcfg.py --oauth2 update api.yaml app.yaml
git push appengine master (configure for automatic deployment)

Is the Queue good?
Mostly not. One way queue.
You just submit the request and cross your fingers.

A bunch of "Messaging Stuff" in the pipeline.

Data center closest to the user will start new machines if neccessary.
Cloud Data Service runs similar for closes geography.
CloudSQL is a much more normal VM of a DB.

Adaptive Delivery - The other responsive - Notes

My Notes From: Adaptive Delivery - The other responsive
https://joind.in/10656

* The Other Responsive

Responsive -> **Client** adapts the site
    CSS(Media Queries) / Javascript / Single Code Base
    More CSS Code that everybody downloads - More Processing time.
    Less Testing / Less Debugging

Adaptive Delivery -> **Server** adapts the site
    Networking to multiple code bases
    Single code base for different content

Device Definition Library
    WURFL (Community Version) -> Plugin to Apache
    Giant XML file with API

Gannett's Solution
    Dual application with shared library of code
    Network level adaptive delivery handled by Akamai (Same URL)

7" Tablets get Mobile Layout
10" Tablets get Desktop Layout

Lower end mobile devices don't support scroll support, but do have touch.
They use a "..." for clicking instead of scrolling.

And... he's done early.

Accelerating Web Applications with Varnish - Notes

My Notes From: Accelerating Web Applications with Varnish
https://joind.in/10648

* About Me
Varnish is the first real revolutionary tech encountered

Static content continues to dominate the web
No One Likes Stale

Dynamic content is expensive to delivery but cheap to maintain

Content Delivery via HTTP (Historical)

Dynamic Backends
Databases! Venerable! Relational!
More and More Complex Databases Start to Be Terrible

SQL Query is Relational Algebra

Cacheing
Eliminate repeated interactions at cost of freshness.
Increases performance of course (throw in front of a DB)
Reddis: Cache and Datastore are starting to merge.
Cacheing proxies

Memcache is developer focused
CDN is not flexible
Static Pre-Rendering
  Just render all files statically and serve those
  Hides terrible performance
Cache Reverse Proxies
  Symfony2 Comes with a Reverse Proxy

Varnish 3 (Varnish 4 exists and is cool/fun)

Virtual Memory is a cache!
Caching-Reverse Proxies.. Squid writes objects to disk, written to memory when requested

Varnish works within the Operating Systems VM.
Trusts the kernel to do the kernel's job.

varnish-cache.org
varnish-software.com

VCL is Varnish Config Language (You can Inline C Programming if you are crazy)
VCL is compiled into memory and can be hard to configure

-f VCL file to load
-s Config storage
-T Where Varnish listens for admin
-a Where Varnish listens for HTTP

Backend - Apache, NGINX, etc.
Can use a cluster of backends.

Logical clusters of backends

Load Balancing to Varnish Server.
Varnish Server to Load Balanacing to Origin Servers.
Round Robin to a bunch of servers.

Directors are rules that Varnish follows so you can tell Varnish how to run

Probes
Kind of Object Oriented VCS. Something about Varnish 4.

ACLs
Bypass the proxy for local clients.

Hooks
Pre-defined points in the request-response cycle.
recv,pipe,pass,hit,miss
fech,backend_fech,

Grace Mode
If it can't access a resouces you can make it return the last available resouce.

Saint Mode
Return expired objects
200/50x

Managing Varnish
Command Line Interface is available

Purging is pretty simple.
PURGE /obsolete/resource
Clears memory
Not instantly happening

Banning Objects
Never delivery from cache, always origin.
Is Instant, but doesn't clear memor.

Vary
gzip/deflate are usually variations of the same thing so no need to try both.

Edge Side Includes
Varnish supports a subset of ESI
esi:include
esi:remove

Replaying Traffic
A log that can be played on cold Varnish to warm it up

Complex
A programming language that you have learn.
Default is an ugly flow chart.
It is C so you can break it easily.

Politico doesn't use memcached now.
All services live behind varnish servers.

Varnish -> Application -> Varnish -> API

VCL has "Objects" that are modular.
Modular Stuff!

Varnish does suport SSL (according to the documenation)

A/B Testing Server Side stuff would be terribly more difficult than the JS stuff we do.

Hacking Sites for Fun and Profit - Notes

My Notes From: Hacking Sites for Fun and Profit

https://joind.in/10664

MailCatcher
SQL Injection

?id=5 and 1=1
?id=5 and 1=2
?id=5 and substring(@@version)

XSS send session to another site.

Filter Input. Escape Output.

Command Injection
Escape Shell Arg

Code, Regex, Log, LDAP Injection

Session Puzzling

** Admin Password From the Forgot Password Form
Forgot Password with Username "admin" and email of registered user.
Emailed to the user without any problems.

** Registration Form
username1').('email', 'fist2', 'last2', 'x', 'pw2', 'username2'
first
last
email
password

** Search Input
Passing the input unescaped into a grep command. *; ls -al
gouda *; cat....

** Dev Mode
http://192.168.33.199/apigility/ui#/

Hackathon On Wednesday
Get involved in OpenSource and a Project

Build your first Symfony2 application - Notes

My Notes From: Build your first Symfony2 application
https://joind.in/10662

** Framework Overview

** History
Mohave
Standard Distribution

** Use Composer
"Doc-Treen"
composer create-project symfony/framework-standard-edition myProject/ 2.4.4

777 app/cache
777 app/logs

app directory is where all the app stuff lives.
bin directory containst a couple binaries
src where all custom code
vendor third party librarires
web is document root

assetic is a thing

use composer again!

config/config.yml
config/routing.yml
config/security.yml

Resources

** Web Directiory
app.php
app_dev.php
bundles

** Bundles
controllers, entities, views, configs, forms, etc.

** Libraries
Composer install should update all the things.

app console generate:bundle
namespace - then Bundle becuse you need to name bundles "Bundle"

He prefers annotation configuration
app/AppKernel.php

routing.yml??

app/console cache:clear
sudo app/console cache:clear

** Doctrine Databases
He's having problems... Ooops.

app/console doctrine:generate:create
app/console doctrine:generate:entity

app/console doctrine:schema:update --dump-sql
app/console doctrine:schema:update --force

Doctrine Migrations??
Not going to show you those right now.

app/console doctrine:generate:crud

Doctrine has a soft delete extension.
Remove removes it from the database hard.

Make sure you have a service layer between your controller and doctrine.
It's the best practice.

** Twig!
Define Blocks

** Forms!
Forms! Objects! Validation!
PostType extends AbstractType

Just reviewing the automatically generated code for form.

Group Validators ??

Propel (http://propelorm.org/) is an alternative to Doctrine.

** Services

Creating a PDO service from scratch.
When creating a PDO service don't register a service for "/PDO" but use "PDO" instead

Never inject the container because it's untestable.

Raspberry Pi as TimeMachine

Install the 2013-12-20 Wheezy Rasbpian disk image on an SD card. This will probably work fine on later versions, but I'm being overly specific.

You'll probably want to install a new user instead of the usual pi/raspberry user so follow the guide I wrote here: Change the Default Pi User.

Install vim because great Caesar's Ghost I can never get vi to work right.

sudo aptitude install vim

 Change your hostname because you don't need any extra devices on your network called "raspberrypi"

sudo vim /etc/hostname
sudo vim /etc/hosts

I called mine "BackupPi" because I'm so clever and interesting.
Install enough tools to allow your Macs to find your Pi.

sudo aptitude install avahi-daemon netatalk

Restart your Pi so that you know nothing went crazy.

sudo shutdown -r now

Connect a hard drive to the USB port formatted as EXT4. I don't care how it is done, just do it. Mine shows up as /dev/sda1 but yours might be different.
Now mount your drive.

sudo mkdir /media/external
sudo chmod 775 /media/external
sudo chgrp pi /media/external
mount /dev/sda1 /media/external

Test it to make sure you have proper access to it.

touch /media/external/test_touch

Okay, good work. Now make sure it mounts on boot.

sudo vim /etc/fstab

Add an additional line to the configurations.

/dev/sda1      /media/external      ext4      defaults,rw      0      0

Restart your Pi so that you know nothing went crazy and your disk will auto mount.

sudo shutdown -r now

After it comes back hopefully you can see what is in your drive.

ls /media/external

If you don't see what you are hoping for something went wrong. Crap. Sorry.
If you get to see the test_touch file you created before, congratulations. You did it!
Now you need to get your Mac to know that what you created is a good spot to put your backups. Make a copy of the original settings and create a new file.

sudo mv /etc/netatalk/AppleVolumes.default /etc/netatalk/AppleVolumes.original
sudo vim /etc/netatalk/AppleVolumes.default

Put the following in the new file:

:DEFAULT: options:upriv,usedots
/media/external "External Storage" options:tm allow:@pi

Save it and restart. Cross your fingers.

sudo shutdown -r now

Once it restarts You should be able to easily see that there is a machine with shares available on my network with the name you chose previously. Mine is "BackupPi" and it has a shared folder of "External Storage" so now just tell Time Machine to backup to that folder.

Helpful Links:
http://d43.me/blog/1660/concisest-guide-to-setting-up-time-machine-server-on-ubuntu-server-12-04/
http://www.techjawab.com/2013/06/how-to-setup-mount-auto-mount-usb-hard.html

Create Your User on Raspberry Pi and Remove the Default

Since Raspbian comes with a standard user/password for all installs it makes some sense to create your own user/password combination in an attempt at some security.

I'm going to be creating the user "spoon" and deleting the user "pi" today.

sudo adduser spoon

This command will prompt you for a lot of information. Fill in as much as you want. The password bit is the only real important part.

You want to make sure the user "spoon" has the same groups as the user "pi" so checkout the groups that pi is in.

groups pi

Now set those same groups to the user spoon. I like to see they are exactly the same so using these commands sets them in the exact same order

sudo usermod -g pi spoon
sudo usermod -G gpio,spi,input,netdev,users,games,plugdev,video,audio,sudo,cdrom,dialout,adm,pi spoon

Now compare them to make sure they are the same. These two commands should look identical.

groups pi
groups spoon

Lastly, log out, login as your new user, verify you have sudo power, and nuke the user "pi" out of the sky.

sudo deluser -remove-home pi