Analyzing the Data

Now for the fun to start! Let’s create our first analysis script. Create a new file in the analysis folder called noise_stats.py. Copy and paste the following code into the file.

Listing 11 noise_stats.py

# import packages
import pandas as pd
from thot import ThotProject

# initialize thot project
db = ThotProject()

# get noise data from asset
noise_data = db.find_asset( { 'type': 'noise-data' } )

# import noise data into a pandas data frame
df = pd.read_csv( noise_data.file, header = 0, index_col = 0, names = [ 'trial', 'volume' ] )

# compute statistics
stats = df.describe()

# create a new asset for the statistics
stats_properties = {
	'name': 'Noise Statistics',
	'type': 'noise-stats',
	'file': 'noise-stats.csv'
}

stats_path = db.add_asset( stats_properties, 'noise_stats' )

# export the statistics to the new asset
stats.to_csv( stats_path ) 

Let’s go through and break down what each chunk of code is doing.

• lines 2-3: Import the packages we’re goin to use, namely Pandas and Thot. In this case we only need to use a small part: ThotProject.

• line 6: Initialize the Thot project, giving us access to all the data stored within it.

• line 9: Find the Noise Data Asset we made for each batch by search for Assets who have a type of ‘noise-data’.

• line 12: Load the noise data into a Pandas DataFrame.

• line 15: Compute statistics on the noise data.

• lines 18-24: Create a new Asset to store the noise statistics in. Notice that the stats_properties dictionary we pass in mimics exactly the structure of the _asset.json files we created earlier. db.add_asset() accepts as its second argument an _id for the new Asset.

• line 27: Saves the statistics to the new Asset.

Now we need to tell Thot which Containers to run this script from. This is done by creating Script Associations.

desktopcli

Again, we’ll see how to add Script Associations from both the Project and File Tree views.

From the Project view right click on Recipe A > Batch 1 and select Edit Scripts. This open the Script Associations dialog. Click the Add Script button at the top of the dialog and select the noise_stats.py Script we just created. We can ensure the Script Association was created successfully by changing the preview from Assets to Scripts.

Fig. 27 Script Associations dialog.

For Recipe A > Batch 2 lets assign the same Script, but instead of right clicking on the Container to open the Script Associations dialog, double click on the Scripts preview of the Container. At the moment this is (none) for the Container.

Now switch to the File Tree view and select Recipe B > Batch 1. Click the Add Scripts button, and perform the same steps.

Add Scripts will add Scripts to the selected Container, while Set Scripts will remove any previously associated Scripts, and set them to match what is submitted.

Let’s run our first analysis! From the Project view switch the Assets preview so we can see our new Assets being created. Then, when you’re ready, click the Analyze button in the upper right of the workspace.

Warning

Running the analysis by pressing the Analyze button may give you an error. If this occurs please attempt to run the analysis from the command line.

To do this open up a terminal (Anaconda prompt on Windows) navigate to teh project root (data folder) and run thot run.

More information is available in the cli tab of this section.

Fig. 28 Analyze button.

When the analysis is running we can continue to work on our project, and when the analysis is complete we will get a notification pop up and the new Assets will appear in our preview.

Click here to download this project step.

Moving On Up

Now that we have the statistics for each of our batches we can move up one level in our project tree to compile the statistics for each recipe. Let’s first make the analysis script calling it recipe_stats.py.

Listing 13 recipe_stats.py

# include packages
import pandas as pd
from thot import ThotProject

# initialize thot
db = ThotProject()

# get recipe container
recipe = db.find_container( { '_id': db.root } )

# get noise statistics data
noise_stats = db.find_assets( { 'type': 'noise-stats' } )

# create combined dataframe 
df = []
for stat in noise_stats:
	# read data for each batch
	tdf = pd.read_csv( 
		stat.file, 
		names = [ stat.metadata[ 'batch' ] ], 
		index_col = 0, 
		header = 0 
	)
	
	df.append( tdf )

df = pd.concat( df, axis = 1 )

# compute recipe statistics
mean = df.loc[ 'mean' ].mean() 
std = df.loc[ 'std' ].pow( 2 ).sum()/ 4 

stats = pd.DataFrame( [ mean, std ], index = ( 'mean', 'std' ) )

# export recipe statistics
stat_properties = {
	'name': '{} Statistics'.format( recipe.name ),
	'type': 'recipe-stats',
	'file': 'recipe-stats.pkl'
}

stats_path = db.add_asset( stat_properties, 'recipe_stats' )
stats.to_pickle( stats_path )

Let’s look at some of the new things we did here:

• line 9: Get the Container the script is running in. In this case it will be Recipe A and Recipe B.

• line 12: When analyzing the batches we only had one Asset we wanted to use, so used the find_asset() method. Now we want to pull in both noise_stats Assets, so use the find_assets() method, which returns a list of Assets that match the criteria. Also notice that the noise_stats aren’t in the Recipe Containers directly, but are in the batch children Containers. This highlights a very important point: Containers have access to their Assets as well as all their childrens’ Assets.

• line 16-23: Iterate over each noise_stats Asset, creating a Pandas DataFrame from it, and adding it to the data list to be combined in line 27.

• line 37: Use the name of the Container as part of the name for the new Asset.

• line 43: Export the new Asset, this time as a pickle (.pkl) file. This is a binary format used by Pandas to store DataFrames, making importing them later on easier.

Let’s add this new script to run on our recipe Containers, and then run it.

desktopcli

Use any of the methods we learned before to associate the recipe_stats.py Script to the Recipe A and Recipe B Containers, then analyze the project again.

Let’s build our final analysis script now so we can see which recipe is better. In the analysis folder create the recipe_comparison.py script.

Listing 14 recipe_comparison.py

# import packages
import pandas as pd
from thot import ThotProject

# intialize thot
db = ThotProject()

# prepare data
recipe_stats = db.find_assets( { 'type': 'recipe-stats' } )

df = []
for stat in recipe_stats:
    # read data for each recipe
    tdf = pd.read_pickle( stat.file )
    tdf.rename( { 0: stat.metadata[ 'recipe' ] }, axis = 1, inplace = True  )
    
    df.append( tdf )

# combine into one dataframe
df = pd.concat( df, axis = 1 )

# export data as csv for reading
comparison_properites = {
	'name': 'Recipe Comparison',
	'type': 'recipe-comparison',
	'file': 'recipe_comparison.csv' 
}

comparison_path = db.add_asset( comparison_properites, 'recipe_comparison' )
df.to_csv( comparison_path )

# create bar char and export
means = df.loc[ 'mean' ]
errs = df.loc[ 'std' ]

ax = means.plot( kind = 'bar', yerr = errs )

bar_properties = {
	'name': 'Recipe Comparison',
	'type': 'recipe-bar-chart',
	'tags': [ 'chart', 'image' ],
	'file': 'recipe_comparison.png'
}

bar_path = db.add_asset( bar_properties, 'recipe_bar' )
ax.get_figure().savefig( bar_path, format = 'png' )

Let’s breakdown the new concepts:

• line 15: Use the recipe metadata from the Asset to name the data.

• line 29, 45: We’ve already seen that we can pull in multiple Assets into our scripts. Here we also see that we can create multiple Assets in a single script. Also notice that one Asset is a CSV text file, while the other is a PNG image file. Assets can be any sort of file.

desktopcli

Add the proper Script Association to the Silent Fireworks Container, and analyze the project.

And we’re done! Take a look at the Assets we created so we know which recipe is quieter and can report to the boss which we should use to keep those fish as happy as possible.

Click here to download the final project.