In this demo, we use PyAirbyte to extract data from Github, followed by a series of transformations and analyses to derive meaningful insights from this data. In particular, we demonstrate PyAirbyte capabilities for extracting data incrementally.
Prerequisites: A Github personal access token. For details on configuring authetication credentials, refer to the Github source connector documentation . # Add virtual environment support in Google ColabLoad source data from Github to local cache In this section, we establish a connection to Github via PyAirbyte. The source connector is configured with necessary parameters like the credentials repository we are interested in. Check the docs for more details on these parameters.
After configuring the source connector, we perform a check() to ensure that the configuration is correct and the connection to the API is successful. Then, we list the available streams for this source and select the ones we are interested in syncing.
Then, we proceed to read from the source into the internal DuckDB cache.
Note: The credentials are retrieved securely using the get_secret() method. This will automatically locate a matching Google Colab secret or environment variable, ensuring they are not hard-coded into the notebook. Make sure to add your key to the Secrets section on the left.
import airbyte as abInstalling 'source-github' into virtual environment '/content/.venv-source-github'.
Connector 'source-github' installed successfully!
Connection check succeeded for `source-github`.
# List the available streams available for the Github source['issue_timeline_events',
# Select the streams we are interested in loading to cacheRead Progress Started reading at 22:16:25.
Read 314 records over 6 seconds (52.3 records / second).
Wrote 314 records over 4 batches.
Finished reading at 22:16:32.
Started finalizing streams at 22:16:32.
Finalized 4 batches over 2 seconds.
Completed 4 out of 4 streams:
reviews stargazers pull_requests issues Completed writing at 22:16:34. Total time elapsed: 8 seconds
Completed `source-github` read operation at 22:16:34 .
Read again to sync changes The PyAirbyte Github source connector has the ability to read data incrementally by default, meaning that after the first successful data sync, only updates and new records will be synced in subsequent reads.
For more information on sync modes for this source, you can refer to the docs .
Let's read again, and see how no records will be loaded to cache.
result = source.read(cache=cache) Read Progress Started reading at 22:17:30.
Read 0 records over 3 seconds (0.0 records / second).
Finished reading at 22:17:33.
Started finalizing streams at 22:17:33.
Finalized 0 batches over 0 seconds.
Completed 4 out of 4 streams:
reviews stargazers pull_requests issues Completed writing at 22:17:33. Total time elapsed: 3 seconds
Completed `source-github` read operation at 22:17:33 .
Read data from the cache Read from the already-written DuckDB cache into a pandas Dataframe. After the data is in the cache, you can read it without re-configuring or re-creating the source object. You can also select a specific stream to read from.
Transform and analyze the data Let's take our Github data and run some analysis with pandas and matplotlib.
repository user_id starred_at userrepository url id node_id html_url diff_url patch_url issue_url commits_url review_comments_url ... assignee assignees requested_reviewers requested_teams head base _links author_association auto_merge draftrepository id node_id url repository_url labels_url comments_url events_url html_url number ... pull_request closed_at created_at updated_at author_association draft reactions timeline_url performed_via_github_app state_reasonPull Requests Trend Over Time This will visualize the number of pull requests created over time.
import pandas as pdOpen vs Closed Issues Compare the number of open versus closed issues.
# Assuming 'issues' dataframe has a 'state' columnStar Growth Over Time Plot the cumulative count of stars over time.
# Assuming 'stargazers' dataframe has a 'starred_at' column in datetime format
Issue Resolution Followed by Pull Requests This analysis checks if there's a trend in issue resolutions followed by pull requests, which might indicate active development in response to issues.
# Convert created_at and closet_at to datetimecombined_issues_prs_df.plot(kind="line", figsize=(12, 6))
About the Author About the Author