![](\"/img/image-1-gcp-billing-1-.jpg\")
For any successful cloud adoption, gaining comprehensive visibility into ongoing cloud spend is essential - no one wants to receive a bill higher than expected when trying to plan a budget.
\nIn the case of Google Cloud Platform, there are different charging models for different resources. For example, GCE cost depends on machine type (based on CPU, memory, network and disks), Google Kubernetes Engine (GKE) and Google Cloud Dataproc charges are based on all running nodes on Google Compute Engine (GCE), while some other service costs have more complex formulas. It becomes increasingly difficult to predict cloud spends, especially when you use a lot of different resources. It is important to be aware of the spending and be able to respond in time if it becomes too expensive. If you can proactively monitor billing reports every day, the probability of receiving a surprising bill at the end of the month is drastically reduced.
\nIn a recent project, we wanted to implement an extended billing report functionality to efficiently track spending. While GCP allows exporting billing data into BigQuery for further comprehensive analysis, manually querying BigQuery tables is unlikely sufficient enough; ideally, an operator is able to visualise data and to filter and aggregate by additional parameters. The most common filters are resource type, time period, team and department, and comparison with previous periods.
\nThe first solution we tried was based on Google's suggestions to use Google Data Studio. While the implementation was very straightforward because of the sample reports and dashboards already available, and only data source configurations were required, the solution was not flexible enough. At creating a chart in Google Data Studio, the user needs to choose all parameters manually instead of providing a formula. To make the billing report more user-friendly we decided to look for another way.
\nBased on our previous experience we thought that Grafana would be a good option to visualise billing data if we could find a way to connect it to BigQuery (BQ). We tried an open-source BQ plugin for Grafana, but it contained a lot of bugs and was not stable enough. Additionally, there was another issue - BQ jobs take too much time to return data.
\nEventually, we decided to load data to PostgreSQL in CloudSQL (because CloudSQL is the easiest way to have a relational database instance) and use Grafana for visualisation. Grafana has an official PostgreSQL plugin, we tested it and realized that it was the best fit to our requirements.
\nThe diagram below is an overview to the workflow we implemented. We deployed a cronjob to the Kubernetes cluster that triggered a Cloud Dataflow job every 4 hours. The dataflow job initiated data loading from BQ to PostgreSQL. It checked max(export_time) in PostgreSQL and loaded data from BQ incrementally, i.e.only importing the data since the last BQ export. The end-users could monitor Grafana dashboards connected to the PostgreSQL instance for the latest billing data.
\n
First, we set up billing data export to BigQuery, as described here.
\nThe easiest way to create a PostgreSQL instance is by using Cloud SQL.
\nAfter the instance was up and running, we created a number of database objects, including a table that mimicked the BQ table structure and a materialized view with indexes that we would use for connecting to PostgreSQL from Grafana.
\nCREATE DATABASE billing;\nUSE billing;\n\nCREATE TABLE public.billing_export_2 (\n id serial NOT NULL,\n sku_id varchar NULL,\n labels varchar NULL,\n export_time varchar NULL,\n currency varchar NULL,\n sku_description varchar NULL,\n location_zone varchar NULL,\n currency_conversion_rate float8 NULL,\n project_labels varchar NULL,\n location_country varchar NULL,\n usage_start_time varchar NULL,\n billing_account_id varchar NULL,\n location_region varchar NULL,\n usage_pricing_unit varchar NULL,\n usage_amount_in_pricing_units float8 NULL,\n cost_type varchar NULL,\n project_id varchar NULL,\n system_labels varchar NULL,\n project_description varchar NULL,\n location_location varchar NULL,\n project_ancestry_numbers varchar NULL,\n credits varchar NULL,\n service_description varchar NULL,\n usage_amount float8 NULL,\n invoice_month varchar NULL,\n usage_unit varchar NULL,\n usage_end_time varchar NULL,\n \"cost\" float8 NULL,\n service_id varchar NULL,\n CONSTRAINT billing_export_2_pkey PRIMARY KEY (id)\n);It is a good practice to add a minimal set of labels to all resources. Every label from this set is represented as a separate column in the view. We also created indexes on these columns to make the Grafana queries faster.
\nCREATE MATERIALIZED VIEW vw_billing_export AS\n SELECT\n id,\n sku_id,\n labels,\n export_time::timestamp,\n currency,\n sku_description,\n location_zone,\n currency_conversion_rate,\n project_labels,\n location_country,\n usage_start_time::timestamp,\n billing_account_id,\n location_region,\n usage_pricing_unit,\n usage_amount_in_pricing_units,\n cost_type, project_id,\n system_labels,\n project_description,\n location_location,\n project_ancestry_numbers,\n credits,\n service_description,\n usage_amount,\n invoice_month,\n usage_unit,\n usage_end_time::timestamp,\n \"cost\",\n service_id,\n l_label1 ->> 'value' as label1,\n l_label2 ->> 'value' as label2,\n ...\n FROM billing_export_2\n LEFT JOIN jsonb_array_elements(labels::jsonb) AS l_label1\non l_label1 ->> 'key' = 'label1'\n LEFT JOIN jsonb_array_elements(labels::jsonb) AS l_label2\non l_label2 ->> 'key' = 'label2'\n ...After that we also created indexes on the view. The indexes could be created later -- it was more important to first understand how the queries would look like before creating the indexes.
\nCREATE INDEX vw_billing_export_label1\n ON vw_billing_export (label1);A service account with access to DataFlow and BigQuery was needed as this was how the DataFlow job would retrieve the BigQuery data.
\nexport project=myproject\ngcloud iam service-accounts create \"bq-to-sql-dataflow\" --project ${project}\n\ngcloud projects add-iam-policy-binding ${project} \\\n--member serviceAccount:\"bq-to-sql-dataflow@${project}.iam.gserviceaccount.com\" \\\n--role roles/dataflow.admin\n\ngcloud projects add-iam-policy-binding ${project} \\\n--member serviceAccount:\"bq-to-sql-dataflow@${project}.iam.gserviceaccount.com\" \\\n--role roles/bigquery.dataViewerCloud DataFlow jobs need 2 buckets to store the temporary and outputs respectively.
\ngsutil mb gs://some-bucket-staging\ngsutil mb gs://some-bucket-tempCloud DataFlow supports Python and Javascript code. In this implementation we used Python. Other than the Apache Beam library, as part of the implementation we needed a JSON file with the service account credentials set up earlier and set to the GOOGLE_APPLICATION_CREDENTIALS environment variable.
For data consistency we defined max(export_time) in PostgreSQL and loaded records from BQ starting from this time.
\nWe also needed a requirements.txt file that contains a list of packages to be installed on workers. In our case we needed only one package beam-nuggets.
\nThis was how the main part of the script (bq-to-sql.py) looked.
args = parser.parse_args()\nproject = args.project\njob_name = args.job_name + str(uuid.uuid4())\nbigquery_source = args.bigquery_source\npostgresql_user = args.postgresql_user\npostgresql_password = args.postgresql_password\npostgresql_host = args.postgresql_host\npostgresql_port = args.postgresql_port\npostgresql_db = args.postgresql_db\npostgresql_table = args.postgresql_table\nstaging_location = args.staging_location\ntemp_location = args.temp_location\nsubnetwork = args.subnetwork\n\noptions = PipelineOptions(\n flags=[\"--requirements_file\", \"/opt/python/requirements.txt\"])\n# For Cloud execution, set the Cloud Platform project, job_name,\n# staging location, temp_location and specify DataflowRunner.\n\ngoogle_cloud_options = options.view_as(GoogleCloudOptions)\ngoogle_cloud_options.project = project\ngoogle_cloud_options.job_name = job_name\ngoogle_cloud_options.staging_location = staging_location\ngoogle_cloud_options.temp_location = temp_location\ngoogle_cloud_options.region = \"us-west1\"\nworker_options = options.view_as(WorkerOptions)\nworker_options.zone = \"us-west1-a\"\nworker_options.subnetwork = subnetwork\nworker_options.max_num_workers = 20\n\noptions.view_as(StandardOptions).runner = 'DataflowRunner'\n\nstart_date = define_start_date()\n\nwith beam.Pipeline(options=options) as p:\n rows = p | 'QueryTableStdSQL' >> beam.io.Read(beam.io.BigQuerySource(\n query='SELECT \\\n billing_account_id, \\\n service.id as service_id, \\\n service.description as service_description, \\\n sku.id as sku_id, \\\n sku.description as sku_description, \\\n usage_start_time, \\\n usage_end_time, \\\n project.id as project_id, \\\n project.name as project_description, \\\n TO_JSON_STRING(project.labels) \\\n as project_labels, \\\n project.ancestry_numbers \\\n as project_ancestry_numbers, \\\n TO_JSON_STRING(labels) as labels, \\\n TO_JSON_STRING(system_labels) as system_labels, \\\n location.location as location_location, \\\n location.country as location_country, \\\n location.region as location_region, \\\n location.zone as location_zone, \\\n export_time, \\\n cost, \\\n currency, \\\n currency_conversion_rate, \\\n usage.amount as usage_amount, \\\n usage.unit as usage_unit, \\\n usage.amount_in_pricing_units as \\\n usage_amount_in_pricing_units, \\\n usage.pricing_unit as usage_pricing_unit, \\\n TO_JSON_STRING(credits) as credits, \\\n invoice.month as invoice_month, \\\n cost_type \\\n FROM `' + project + '.' + bigquery_source + '` \\\n WHERE export_time >= \"' + start_date + '\"',\n use_standard_sql=True))\n source_config = relational_db.SourceConfiguration(\n drivername='postgresql+pg8000',\n host=postgresql_host,\n port=postgresql_port,\n username=postgresql_user,\n password=postgresql_password,\n database=postgresql_db,\n create_if_missing=True,\n )\n table_config = relational_db.TableConfiguration(\n name=postgresql_table,\n create_if_missing=True\n )\n rows | 'Writing to DB' >> relational_db.Write(\n source_config=source_config,\n table_config=table_config\n )After the data was loaded we needed to refresh the materialized view. Since normally the refresh would take some time, it was also possible to create a new materialized view with the same structure (and the corresponding indexes), delete the old view and rename the new one to the old name.
\nThe same service account used earlier in the Cloud Dataflow workflow is also used in the cron job. The following command created a private key of the service account, that we subsequently uploaded to the Kubernetes cluster as secret to be accessed in the cron job.
\ngcloud iam service-accounts keys create ./cloud-sa.json \\\n--iam-account \"bq-to-sql-dataflow@${project}.iam.gserviceaccount.com\" \\\n--project ${project}kubectl create secret generic bq-to-sql-creds --from-file=./cloud-sa.jsonWe wanted the DataFlow job to run on a daily basis. First of all, we created a Docker image with all the needed environment variables and the Python script.
\nTo create a file with commands to run:
\n#!/bin/bash -x\n\n#main.sh\n\nproject=${1}\njob_name=${2}\nbigquery_source=${3}\npostgresql_user=${4}\npostgresql_password=${5}\npostgresql_host=${6}\npostgresql_port=${7}\npostgresql_db=${8}\npostgresql_table=${9}\nstaging_location=${10}\ntemp_location=${11}\nsubnetwork=${12}\n\nsource temp-python/bin/activate\n\npython2 /opt/python/bq-to-sql.py \\\n --project $project \\\n --job_name $job_name \\\n --bigquery_source $bigquery_source \\\n --postgresql_user $postgresql_user \\\n --postgresql_password $postgresql_password \\\n --postgresql_host $postgresql_host \\\n --postgresql_port $postgresql_port \\\n --postgresql_db $postgresql_db \\\n --postgresql_table $postgresql_table \\\n --staging_location $staging_location \\\n --temp_location $temp_location \\\n --subnetwork $subnetworkThe content of the Dockerfile
\nFROM python:latest\nRUN \\\n bin/bash -c \" \\\n apt-get update && \\\n apt-get install python2.7-dev -y && \\\n pip install virtualenv && \\\n virtualenv -p /usr/bin/python2.7 --distribute temp-python && \\\n source temp-python/bin/activate && \\\n pip2 install --upgrade setuptools && \\\n pip2 install pip==9.0.3 && \\\n pip2 install requests && \\\n pip2 install Cython && \\\n pip2 install apache_beam && \\\n pip2 install apache_beam[gcp] && \\\n pip2 install beam-nuggets && \\\n pip2 install psycopg2-binary && \\\n pip2 install uuid\"\n\nCOPY ./bq-to-sql.py /opt/python/bq-to-sql.py\nCOPY ./requirements.txt /opt/python/requirements.txt\nCOPY ./main.sh /opt/python/main.sh\n\nFROM python:latest\nRUN \\\n bin/bash -c \" \\\n apt-get update && \\\n apt-get install python2.7-dev -y && \\\n pip install virtualenv && \\\n virtualenv -p /usr/bin/python2.7 --distribute temp-python && \\\n source temp-python/bin/activate && \\\n pip2 install --upgrade setuptools && \\\n pip2 install pip==9.0.3 && \\\n pip2 install requests && \\\n pip2 install Cython && \\\n pip2 install apache_beam && \\\n pip2 install apache_beam[gcp] && \\\n pip2 install beam-nuggets && \\\n pip2 install psycopg2-binary && \\\n pip2 install uuid\"\n\nCOPY ./bq-to-sql.py /opt/python/bq-to-sql.py\nCOPY ./requirements.txt /opt/python/requirements.txt\nCOPY ./main.sh /opt/python/main.sh\n\nimage:\nimageTag: latest\nimagePullPolicy: IfNotPresent\nproject:\njob_name: \"bq-to-sql\"\nbigquery_source: \"[dataset].[table]\"\npostgresql:\n user:\n password:\n host:\n port: \"5432\"\n db: \"billing\"\n table: \"billing_export\"\nstaging_location: \"gs://my-bucket-stg\"\ntemp_location: \"gs://my-bucket-tmp\"\nsubnetwork: \"regions/us-west1/subnetworks/default\"cronjob.yaml
\napiVersion: batch/v1beta1\nkind: CronJob\nmetadata:\n name: {{ template \"bq-to-sql.fullname\" . }}\nspec:\n schedule: \"0 0 * * *\"\n jobTemplate:\n spec:\n template:\n spec:\n restartPolicy: OnFailure\n containers:\n - name: {{ template \"bq-to-sql.name\" . }}\n image: \"{{ .Values.image }}:{{ .Values.imageTag }}\"\n imagePullPolicy: \"{{ .Values.imagePullPolicy }}\"\n command: [ \"/bin/bash\", \"-c\", \"bash /opt/python/main.sh \\\n {{ .Values.project }} \\\n {{ .Values.job_name }} \\\n {{ .Values.bigquery_source }} \\\n {{ .Values.postgresql.user }} \\\n {{ .Values.postgresql.password }} \\\n {{ .Values.postgresql.host }} \\\n {{ .Values.postgresql.port }} \\\n {{ .Values.postgresql.db }} \\\n {{ .Values.postgresql.table }} \\\n {{ .Values.staging_location }} \\\n {{ .Values.temp_location }} \\\n {{ .Values.subnetwork }}\"]\n volumeMounts:\n - name: creds\n mountPath: /root/.config/gcloud\n readOnly: true\n env:\n - name: GOOGLE_APPLICATION_CREDENTIALS\n value: /root/.config/gcloud/creds.json\n volumes:\n - name: creds\n secret:\n secretName: bq-to-sql-credsIn this case, we used Helm for the cron job deployment for ease of deployment and reuse.
\nThe last step was the creation of the Grafana dashboards. While we could use any charts that made sense, we used Data Studio Billing Report Demo as inspiration (of course we had to write all SQL queries from scratch).
\nUsing a separate user in the database with read-only access to the view with billing data and to connect from Grafana is recommended as a security best practice.
\nHere are some examples of charts.
\n![](\"/img/image-2-gcp-billing.jpg\")
![](\"/img/image-3-gcp-billing-1-.jpg\")
![](\"/img/image-4-gcp-billing-1-.jpg\")
The above is a walkthrough of how we implemented a workflow to visualize billing data. The mechanism also supports adding new filters and metrics. At implementation roll-out, the client got a set of useful and fast dashboards that help to monitor Google Cloud spending. With this insight, the client is now empowered to make further cost optimizations.
\n
\nWritten by Dariia Vasilenko
\nHow do manage your cloud spending? Are you wondering if you have workloads that can be optimized? Please let us know at info@myops.co.il. We'd love to hear from you!