Automation with Python

It has been awhile since my last blog post as I’ve been busy learning a new skill. I’m a digital experience account manager, and my job is to help my clients succeed in acquiring direct hotel bookings (and ancillary revenue) via the digital space. Comparing with the print media, the main difference working in the digital world is that most of the tools we’re using involves digital data (of course!). As a strong believer of “do once and use forever,” I’m constantly looking for ways to automate things I do often. Naturally, I’ve started to learn programming again.

Web Scraping with Python: Collecting Data from the Modern Web by Ryan MitchellLearning to program is nothing new to me. I’ve started my career as developer, coding with Java, HTML, CSS and eventually PHP many years back before I became a project manager. Learning a new programming language, Python, is relatively easy; all I need is a couple Python guide-book and Google. The book which started my journey is Web Scraping with Python: Collecting Data from the Modern Web by Ryan Mitchell.

With what I learned in the book, I started learning simple Python programming, setting up Git version control, how to crawl through web pages, connect to APIs and storing the data into databases. I manage to set up a low-end server on Amazon Web Services (AWS) to automate python scripts. Some scripts that I manage to set up includes:

  • Crawling through a website and list down all unique pages the crawler could find. This is a useful script for me if a client came to me and asked me to quote for a project to do something on all the pages on a website. This script allows me to crawl through the website quickly to get a sense of how many pages are there in the website.
  • Weather Forecast – I manage to connect into API to retrieve the 2 hours weather forecast. I setup an automated script to check the weather every hour and send a tweet via my twitter account if showers is happening near the area I’m staying. Also, I’m storing daily weather forecast into MySQL database. I haven’t exactly figured out what I’m going to do with this data but I’m guessing I could figure out if global warming is happening. 🙂

  • Website Down SMS NotificationsWebsite down – Handling websites are part of my daily job. The worst thing that could happen is when a website becomes inaccessible (website down) and nobody is aware of it. When a hotel website is down for an hour, the amount of booking revenue losses is quite intimidating. Thus, I’ve created a script to check on a few websites, in an interval of 30 minutes. Should any of the websites I’m tracking became inaccessible, the script would send SMS to me, via AWS’s Simple Notification System, Interestingly, a high-profile website went down only a few days after I setup this automated script. I was able to react way before client is aware of the situation.
  • Email report – In a digital marketing campaign, which may include search engine optimization, pay-per-click and online media, an account manager would have to communicate the campaign’s results to client via monthly reports. Although the reporting team generates professional campaign reports , we as account managers would usually include additional insight/summary to make the email report concise and easily “digestible” by our clients. This also means that I have to use Google Analytics, do number crunching, summarize data and eventually adding our suggestions and comments. This is fine until the number accounts increase to a point where I could easily spend one whole week doing reporting. I’m hoping to automate the process of consolidating data and number crunching.With the help of Google Analytics API, I created a script to help me retrieve data I need from GA accounts / views, do calculations and comparison and spit out the information that I could easily copy and paste into an email report. I estimated that this script could help me/us saves up to 20 minutes per email report, multiply by the number of accounts my colleagues and I are handling.Email Report

Published by

Johnny Koh

A project manager and a brand strategist, I have over six years of experience in advising businesses of various industries as well as the public sector in their branding and communications need.