Skip to content

feat: Add SensitiveColumns to tables schema #2134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

blesniewski
Copy link
Contributor


Use the following steps to ensure your PR is ready to be reviewed

  • Read the contribution guidelines πŸ§‘β€πŸŽ“
  • Run go fmt to format your code πŸ–Š
  • Lint your changes via golangci-lint run 🚨 (install golangci-lint here)
  • Update or add tests πŸ§ͺ
  • Ensure the status checks below are successful βœ…

@github-actions github-actions bot added the feat label Apr 29, 2025
@blesniewski
Copy link
Contributor Author

Needs a cloudquery-api-go merge first

@murarustefaan
Copy link
Contributor

Does this work with:

  • nested fields? JSONPath, basically? we should probably have a solution for that as well (say .spec.containers[0].env in the kubernetes_pods table). If not doable, than we can retrhink the approach of using the transformer and do table_options πŸ€”
  • can we somehow set these in the arrow schema as well? so the transformer would know of all the sensitive columns and redact them by default with no configuration whatsoever?

@blesniewski
Copy link
Contributor Author

blesniewski commented May 2, 2025

nested fields? JSONPath

This is analogical to PermissionsNeeded, meaning it could contain whatever strings we need for each table, then it's on the transformer side to handle the more complicated cases, which I believe it does as per the docs https://hub.cloudquery.io/plugins/transformer/cloudquery/basic/latest/docs

can we somehow set these in the arrow schema as well?

This we'd have to investigate

Edit:
From my local tests, everything should work as above, meaning we should be able to read from the arrow schema in the basic transformer, and basing on that redact what's marked on table definition

@blesniewski blesniewski marked this pull request as ready for review May 7, 2025 15:44
@blesniewski blesniewski requested a review from a team as a code owner May 7, 2025 15:44
@blesniewski blesniewski requested a review from przste-go May 7, 2025 15:44
@kodiakhq kodiakhq bot merged commit e95674f into main May 9, 2025
10 checks passed
@kodiakhq kodiakhq bot deleted the feature/eng-1033-allow-specifying-sensitive-table-columns-in-the-sdk branch May 9, 2025 10:18
kodiakhq bot pushed a commit that referenced this pull request May 9, 2025
πŸ€– I have created a release *beep* *boop*
---


## [4.80.0](v4.79.1...v4.80.0) (2025-05-09)


### Features

* Add SensitiveColumns to tables schema ([#2134](#2134)) ([e95674f](e95674f))


### Bug Fixes

* **deps:** Update module github.com/cloudquery/cloudquery-api-go to v1.13.9 ([#2143](#2143)) ([77d4b9b](77d4b9b))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants