2015-03-16

Anecdote: That time I had an idea that contributed to the game design

I used to work at Pipeworks Software. We did really cool things like the original XBox demos. I didn't work on Desk Toys, but I did most all of the Ping Pong Ball and Mousetrap room. I did the butterfly flight in Butterfly Garden. I also did part of the boot screen that was used for the original generation XBox. I did the fog, camera paths, and blobby goo.

For Godzilla: Destroy All Monsters Melee" I worked on special effects, rigid body physics, &c. At one point I was coding the results of a monster being thrown into a building. Whenever you destroy a building it is supposed to make the people a bit more angry at you. Unfortunately, when a monster was thrown into a building we had no record of who threw the monster into it. As I contemplated how much work would need to be done to track that information I realized that I could just attribute the destruction to the monster that was thrown. After all, if someone throws me onto an ant pile, the ants are going to get mad at me.

I walked over to a designer and ran the idea past him. It would let the player force some human anger onto their opponent. This wasn't a big deal, but could affect the game as the humans would attack the monster that was doing the most damage to their city. Humans could never kill you, but they could take you down to about 10% health through machine gun fire from helicopters. It seemed like the change would work okay, so I went back and implemented it.

That saved me a few minutes of time and some added complexity to the data structures. And I thought it was a nice feature.

2015-03-11

New Page: My Irrigate Project

I just added a page to the blog introducing my major side project. I've built an Arduino based flow sensor that monitors my irrigation water usage. (It also keeps an eye on my automatic pool filler.) It mails me graphs of water usage every morning so that I can easily see if a sprinkler head has been broken off, thereby saving me money and worry. It also emails me if flow stays on for longer than a set time (in case I've forgotten to turn off water filling the pool). I can check the flow rate in real time from a web page.

I haven't yet packaged up the source, but I'll do that soon. Check it out and leave me comments.


2015-03-09

Xmldump: Dumping Django data state to XML

The problem: Migrating Django model data across model versions

I've been working on learning Django. There's probably a better way of doing this that I don't know about, but as near as I can tell there is no way to back up the state of the db in any way that can migrate across versions. There is

There is a stack overflow question on the topic here. It is an old question, with a few third party solutions. A more recent answer references new Django support for migrations. However, migrations requires that you write significant code for non-trivial changes.

I wanted a solution that would let my classes load themselves from older representations. Preferably automatically, but with the option of custom code for non-trivial cases. I searched around a while and could not find anything, so I wrote it myself.

XMLDump

An example project that illustrates using xmldump is here. The only file you need to use it yourself is serializable.py which defines a mixin. Include it as a base class for any models that you want to be able to dump to xml. The file also includes documentation on how to use it, which I've include that below.


The serializable module can save django data to xml format and
reimport. The intent is that in most cases no customization should
be required. However, I expect that there are still many cases where
the code is not yet able to handle the model relationships.
For instance, I have made no particular effort to implement many to
many relationships.
To generate xml from the models currently in memory:
      import serializable
      xml = serializable.models_to_xml([TopLevelModelClass1, Class2])

The xml that is returned is an instance of the xml.etree.ElmentTree.Element
class. To display it in a readable format:
      from xml.dom.minidom import parseString
      doc = parseString( xml.etree.ElmentTree.tostring(xml) )
      xml_string = doc.toprettyxml('  ')

In order to delete all the data in memory (in preparation for reloading it,
presumably):
      delete_all_models_in_db([TopLevelModelClass1, Class2])

The list of class names that must be passed to the functions need not
contain all the model classes. They should contain the highest level
classes only. The module will discover contained classes, whether they
be models which are referred to by a foreign key, or models which refer
to the high level model class through a foreign key.
However, it may be advisable to overload the "owned_models" function
in order to restrict the default discovery. In some cases it makes
for a nicer xml nesting if references are not followed all the way
down.

Example output

<?xml version="1.0" ?>
<ModelData>
  <xmldump.models.Menu>
    <id type="int">1</id>
    <name type="unicode">Breakfast</name>
    <___owned>
      <xmldump.models.MenuItem>
        <menu to_type="xmldump.models.Menu" type="reference">1</menu>
        <price type="float">4.0</price>
        <id type="int">1</id>
        <name type="unicode">Spam and Eggs</name>
      </xmldump.models.MenuItem>
      <xmldump.models.MenuItem>
        <menu to_type="xmldump.models.Menu" type="reference">1</menu>
        <price type="float">4.5</price>
        <id type="int">2</id>
        <name type="unicode">Eggs and Spam</name>
      </xmldump.models.MenuItem>
      <xmldump.models.MenuItem>
        <menu to_type="xmldump.models.Menu" type="reference">1</menu>
        <price type="float">5.0</price>
        <id type="int">3</id>
        <name type="unicode">Spammity Spam</name>
      </xmldump.models.MenuItem>
      <xmldump.models.MenuItem>
        <menu to_type="xmldump.models.Menu" type="reference">1</menu>
        <price type="float">3.0</price>
        <id type="int">4</id>
        <name type="unicode">Spam</name>
      </xmldump.models.MenuItem>
    </___owned>
  </xmldump.models.Menu>
  <xmldump.models.Order>
    <customer type="unicode">Brian</customer>
    <date type="date">2015-03-04</date>
    <id type="int">1</id>
    <___owned>
      <xmldump.models.OrderEntry>
        <order to_type="xmldump.models.Order" type="reference">1</order>
        <count type="int">1</count>
        <menuitem to_type="xmldump.models.MenuItem" type="reference">1</menuitem>
        <id type="int">1</id>
      </xmldump.models.OrderEntry>
      <xmldump.models.OrderEntry>
        <order to_type="xmldump.models.Order" type="reference">1</order>
        <count type="int">1</count>
        <menuitem to_type="xmldump.models.MenuItem" type="reference">2</menuitem>
        <id type="int">2</id>
      </xmldump.models.OrderEntry>
      <xmldump.models.OrderEntry>
        <order to_type="xmldump.models.Order" type="reference">1</order>
        <count type="int">2</count>
        <menuitem to_type="xmldump.models.MenuItem" type="reference">4</menuitem>
        <id type="int">3</id>
      </xmldump.models.OrderEntry>
    </___owned>
  </xmldump.models.Order>
</ModelData>


(Thanks to Free Online XML Escape Tool - freeformatter.com.)

2015-03-04

Date Time and Datetime conversions

Date Time, Timestamp and Datetime conversions

Python has a great class for storing and manipulating the date and time: datetime.

Unfortunately, it is not the only way to refer to a point in time. Sometimes you need to convert between the two and I frequently forget how to do it.

Table of Conversions

Here is a table of conversions. Pick the row corresponding to the form of time that you have (from the left), and them pick the column corresponding to the form that you would like (from the top). The intersecting cell is the code snippet to get you there.

I have some measure of confidence in the correctness since the table, indeed, this entire post, was created by a script which tested the code shown in the table and lists below. That script is available here. However, I expect that many of the conversions can be improved upon and many may be subtly, or not so subtly, incorrect. Please let me know if you have an improvement.

  • "dt" represents an input datetime.
  • "t" represents an input time struct.
  • "secs" represents an input timestamp (seconds since the Epoch GMT).
  • "localtz" represents the local timezone. This is necessary to account for functions or formats that convert to or from the local timezone.
  • "othertz" represents a timezone that you want to to have. It can be UTC or any other pytz timezone.
  • You need to import "pytz" for timezones.
  • You need to import the "datetime" class from the the datetime module.
  • Note that time structs can not store time at a resolution finer than a second.

I'm sorry, but this table just isn't going to work on mobile. Below is a breakout by the type of representation that you are converting to.

Table of Conversions between various date and time formats
To
Timezone Naive Datetime Local
To
Timezone Naive Datetime UTC
To
Timezone Aware Datetime
To
Time Struct Local
To
Time Struct UTC
To
Timestamp
Build as of now datetime . now ( ) datetime . utcnow ( ) localtz . localize ( datetime . now ( ) ) . astimezone ( othertz ) time . localtime ( ) time . gmtime ( ) time . time ( )
From
Timezone Naive Datetime Local
localtz . localize ( dt ) . astimezone ( pytz . utc ) . replace ( tzinfo = None ) localtz . localize ( dt ) . astimezone ( othertz ) dt . timetuple ( ) localtz . localize ( dt ) . utctimetuple ( ) time . mktime ( dt . timetuple ( ) ) + 1e-6 * dt . microsecond
From
Timezone Naive Datetime UTC
pytz . utc . localize ( dt ) . astimezone ( localtz ) . replace ( tzinfo = None ) pytz . utc . localize ( dt ) . astimezone ( othertz ) pytz . utc . localize ( dt ) . astimezone ( localtz ) . timetuple ( ) dt . utctimetuple ( ) calendar . timegm ( pytz . utc . localize ( dt ) . timetuple ( ) ) + 1e-6 * dt . microsecond
From
Timezone Aware Datetime
dt . astimezone ( localtz ) . replace ( tzinfo = None ) dt . astimezone ( pytz . utc ) . replace ( tzinfo = None ) dt . astimezone ( localtz ) . timetuple ( ) dt . utctimetuple ( ) calendar . timegm ( dt . utctimetuple ( ) ) + 1e-6 * dt . microsecond
From
Time Struct Local
datetime ( *t[:6] ) datetime . utcfromtimestamp ( time . mktime ( t ) ) localtz . localize ( datetime ( *t[:6] ) ) . astimezone ( othertz ) time . gmtime ( time . mktime ( t ) ) time . mktime ( t )
From
Time Struct UTC
pytz . utc . localize ( datetime ( *t[:6] ) ) . astimezone ( localtz ) . replace ( tzinfo = None ) datetime . utcfromtimestamp ( calendar . timegm ( t ) ) pytz . utc . localize ( datetime ( *t[:6] ) ) . astimezone ( othertz ) time . localtime ( calendar . timegm ( t ) ) calendar . timegm ( t )
From
Timestamp
datetime . fromtimestamp ( secs ) datetime . utcfromtimestamp ( secs ) localtz . localize ( datetime . fromtimestamp ( secs ) ) time . localtime ( secs ) time . gmtime ( secs )

List of Specific Conversions

Timezone Naive Datetime Local
From To Timezone Naive Datetime Local
Timezone Naive Datetime UTC pytz.utc.localize(dt).astimezone(localtz).replace(tzinfo=None)
Timezone Aware Datetime dt.astimezone(localtz).replace(tzinfo=None)
Time Struct Local datetime(*t[:6])
Time Struct UTC pytz.utc.localize(datetime(*t[:6])).astimezone(localtz).replace(tzinfo=None)
Timestamp datetime.fromtimestamp(secs)

To build as of now: datetime.now()


Timezone Naive Datetime UTC
From To Timezone Naive Datetime UTC
Timezone Naive Datetime Local localtz.localize(dt).astimezone(pytz.utc).replace(tzinfo=None)
Timezone Aware Datetime dt.astimezone(pytz.utc).replace(tzinfo=None)
Time Struct Local datetime.utcfromtimestamp(time.mktime(t))
Time Struct UTC datetime.utcfromtimestamp(calendar.timegm(t))
Timestamp datetime.utcfromtimestamp(secs)

To build as of now: datetime.utcnow()


Timezone Aware Datetime
From To Timezone Aware Datetime
Timezone Naive Datetime Local localtz.localize(dt).astimezone(othertz)
Timezone Naive Datetime UTC pytz.utc.localize(dt).astimezone(othertz)
Time Struct Local localtz.localize(datetime(*t[:6])).astimezone(othertz)
Time Struct UTC pytz.utc.localize(datetime(*t[:6])).astimezone(othertz)
Timestamp localtz.localize(datetime.fromtimestamp(secs))

To build as of now: localtz.localize(datetime.now()).astimezone(othertz)


Time Struct Local
From To Time Struct Local
Timezone Naive Datetime Local dt.timetuple()
Timezone Naive Datetime UTC pytz.utc.localize(dt).astimezone(localtz).timetuple()
Timezone Aware Datetime dt.astimezone(localtz).timetuple()
Time Struct UTC time.localtime(calendar.timegm(t))
Timestamp time.localtime(secs)

To build as of now: time.localtime()


Time Struct UTC
From To Time Struct UTC
Timezone Naive Datetime Local localtz.localize(dt).utctimetuple()
Timezone Naive Datetime UTC dt.utctimetuple()
Timezone Aware Datetime dt.utctimetuple()
Time Struct Local time.gmtime(time.mktime(t))
Timestamp time.gmtime(secs)

To build as of now: time.gmtime()


Timestamp
From To Timestamp
Timezone Naive Datetime Local time.mktime(dt.timetuple())+1e-6*dt.microsecond
Timezone Naive Datetime UTC calendar.timegm(pytz.utc.localize(dt).timetuple())+1e-6*dt.microsecond
Timezone Aware Datetime calendar.timegm(dt.utctimetuple())+1e-6*dt.microsecond
Time Struct Local time.mktime(t)
Time Struct UTC calendar.timegm(t)

To build as of now: time.time()


Timezones

import pytz, datetime
pytz.timezone('UTC').localize(datetime.utcnow())
pytz.timezone('US/Central').localize(datetime.now())
print 'Common timezones:',pytz.common_timezones

References