Big Data

Last updated on 2026-07-01 | Edit this page

Overview

Questions

  • What does Big Data mean?
  • What are some common barriers to working with big data?

Objectives

  • Recognise that “big data” is context-dependent (size, complexity, velocity).
  • Identify when your own research crosses into “big data” territory.
  • Identify common technical and practical barriers (compute, storage, transfer, tooling).
  • Reflect on which barriers are most relevant to your own research context.

Background

In this workshop we will be returning to the following flow chart as an example of a generic workflow for research:

flowchart LR
    accTitle: {A generic research workflow}
    accDescr: {A generic research workflow with 4 main sections: obtaining data, preparing data, analyzing data, and communicating results.}
    subgraph obtain ["Obtain Data"]
        direction LR
        a["observe"]
        b["simulate"]
        c["literature search"]
        d["download data"]
        e["..."]
    end
    subgraph prepare ["Prepare Data"]
        direction LR
        f["clean"]
        g["filter"]
        h["aggregate"]
        i["classify"]
        j["process"]
        k["..."]
    end
    subgraph analyse ["Analyze Data"]
        direction LR
        l["visualise"]
        m["describe"]
        n["model"]
        o["test hypothesis"]
        p["draw conclusions"]
        q["..."]
    end
    subgraph publish ["Communicate Results"]
        direction LR
        r["article / paper"]
        s["methods / code"]
        t["evidence / data"]
        u["..."]
    end
    obtain --> prepare --> analyse --> publish

The workflow above is intentionally generic and hopefully you have experience in doing many of the activities described. In your research some or all of the above will be done “by hand” (that is, interactively), at least initially, using existing tools and your local machine (your laptop or work desktop).

The advent of “Big Data” in astronomy hasn’t fundamentally changed what the above workflow looks like, however it does change how each of the steps are preformed. This will be the focus of today’s workshop:

  1. What is big data, and how does it change how I do research?
  2. How can we better design, execute, and evaluate workflows?
  3. How do we maintain research best practice when working with big data?

Big Data in Astronomy: Scale, Barriers, and Implications

What is big data?

Despite the name, big data is not all about the petabytes. In fact, the “big” in big data is more about the scale of the problems caused, than the size of the data itself.

Big data begins when your normal way of working breaks.

Essentially you know you need to engage in big data thinking when your established workflows break. Sometimes the solutions require new hardware or software, but sometimes you also need to change how you think about a problem or even change the questions that you are asking. Your workflows may need changing or updating when any of the items listed in our example workflow become difficult. For example, our normal ways of working could break because:

  1. The data are too large to store on your local machine,
  2. The cleaning/filtering process takes too long to complete,
  3. The data have high dimensionality or connectivity, and is difficult to summarize or visualise with existing tools,
  4. Your data can’t be presented as a table or image, and thus is difficult to share in a publication.

But I can get around these problems…

These problems often have simple solutions—but each comes with a cost:

  1. Use less data: You work on a subset but miss subtle effects and potential discoveries.
  2. Apply standard pipelines once: You fix issues later with ad‑hoc corrections and it becomes hard to separate real signals from artefacts.
  3. Reduce complexity: You analyse only a few features and miss important relationships.
  4. Limit sharing You publish snapshots, not data, so results are hard to verify or reuse (and easily lost).

These approaches avoid big data challenges by accepting limitations.

Big data thinking doesn’t remove trade-offs, it gives you better ones.

What actually goes wrong?

Working with big data is not just difficult, it is different. The challenge is not that tasks become slower, but that some things stop being possible altogether. The great news is that the converse is often also true - if you can solve some big data problems, you start to unlock new capabilities.

When you have big data problems the following things happen:

  1. You can’t open or inspect your data
    • Files are too large. Data are distributed.
    • You cannot “just take a look”.
  2. You can’t iterate quickly
    • A small change might require hours or days to re-run.
    • Exploration becomes expensive.
  3. You can’t rerun your analysis reliably
    • Pipelines become complex, fragile, and hard to reproduce.
  4. You can’t keep everything
    • Intermediate results, temporary files, and even raw data may be discarded or inaccessible.

With big data, you stop interacting with the data directly. Instead, you are interacting with the process.

Our new science workflow now looks like this:


flowchart LR;
    accTitle: {Our new science workflow.}
    accDescr: {An alternative science workflow consisting of data input, a fully automated workflow which out puts our results.}
    A["Data"];
    workflow["(Automated) Workflow"];
    R["Results"];
    A-->workflow-->R;

We have taken two steps forward in that we now have an automated and hopefully robust and reproducible workflow that we can rely on. However we also have taken a step back in that we are now less directly working with the data at each stage.

Callout

AT20G: A big data problem without “big data”

The AT20G survey used a new wide-band correlator that enabled much faster observations by increasing bandwidth significantly. Although the total data volume was not unusually large, the project still required big data thinking because it broke existing ways of working:

  • Standard calibration and imaging methods no longer worked.
    • The data did not meet the assumptions of existing software, requiring new processing approaches.
  • Existing tools were insufficient
    • Bespoke software had to be developed to process and interpret the data.
  • Observing strategies had to change
    • Traditional observing methods were replaced with new techniques to handle how the data were collected.
  • Results were harder to validate and communicate
    • Non-standard data products made interpretation and sharing more difficult.

Common big data patterns in astronomy

Modern astronomy is a data-intensive science. In many ways astronomy is leading other domains in it’s embrace of new technologies and techniques thanks to telescopes and simulations that produce data at scales and rates that fundamentally change how research is done. Some common big data patterns that have been adopted in astronomy are:

  1. Streaming data (eg, time-domain astronomy)
    • Data arrives continuously.
    • Decisions must be made in real time.
    • You cannot store everything.
    • You cannot inspect everything by hand.
  2. Data that cannot be downloaded
    • Many datasets are simply too large to move.
    • Analysis happens where the data lives.
    • Attach an HPC to an archive, rather than the inverse.
  3. Pipelines as the primary interface
    • The raw data are rarely used directly.
    • Complex pipelines produce calibrated ‘science ready’ data products.
  4. Long-lived datasets and data releases
    • Results depend on which version of the data you used, and how it was processed.
    • Data and software versioning becomes very important for reproducibility.

As a result we have a frequent data access pattern, where the user is far from the raw data and will often only retrieve a filtered subset of the processed data for their particular research need.


flowchart LR;
    accTitle: {The increased distance between raw data and user accessible data.}
    accDescr: {Users no longer have access to the most raw data products, but instead rely on a pre-processing system and online archive to access data.}
    Observation --> Pipeline --> Archive --> Platform --> User;
    Simulation --> Pipeline;

Why this matters to you

You may not be working with petabytes of data. But you are likely already encountering the same underlying problems:

  • combining multiple datasets
  • re-running analyses late in your project
  • scaling an approach that worked on a small sample
Discussion

Big data projects

Each group has been given a selection of case studies drawn from Astronomy research past and present.

Choose a case study and:

  1. Identify the key pressure point in this project: Where does scale, complexity, or time pressure create difficulty?

  2. If nothing changes, what fails first? Choose one:

    • Storage
    • Compute
    • Data transfer
    • Visualisation
    • Workflow/process
    • Communication/coordination

    Be prepared to justify your choice.

  3. What change would you make to address this? You can change:

    • The data (what is stored, reduced, or discarded)
    • The workflow (timing, automation, decision-making)
    • The tools (algorithms, infrastructure)
    • The people/process (roles, communication)

  4. What new trade-off does your solution introduce? (What gets worse when your fix is applied?)

Join the shared GoogleDoc, locate the tab relevant to the case study you have worked on and record your answers.

If you have time, feel free to complete the above for multiple different projects.