Обсуждение: Permission to allow testing harness to send error reports for pgweb directly to mailing list.
Permission to allow testing harness to send error reports for pgweb directly to mailing list.
От
Akshat Jaimini
Дата:
Hello everyone,
The testing harness has been successfull at catching multiple issues in pgweb in the past few months. The reports are for now being sent to me via email in case of any failure.
Is it ok if I can setup the harness to send the error reports directly to the mailing list?
Regards,
Akshat Jaimini
Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list.
От
Magnus Hagander
Дата:
On Tue, Oct 3, 2023 at 8:24 AM Akshat Jaimini <destrex271@gmail.com> wrote: > > Hello everyone, > The testing harness has been successfull at catching multiple issues in pgweb in the past few months. The reports are fornow being sent to me via email in case of any failure. > > Is it ok if I can setup the harness to send the error reports directly to the mailing list? Hi! I think that would be reasonable, *if* it is ensured that a report is only sent once. That is, if it finds the same issue on a later run, it must not re-send the same thing. How does it work in regards to that today? -- Magnus Hagander Me: https://www.hagander.net/ Work: https://www.redpill-linpro.com/
Hello,
> That is, if it finds the same issue on a later run, it must not re-send the same thing. How does it work in regards to that today?
As per the current flow whenever a new commit is pushed to the pgweb repo, the tests are executed. If some tests fail, an error report is sent with the information of all the failed tests. So if that particular issue has been resolved, the same report won't be sent but if some other commit is pushed without resolving that particular issue then that particular error will be reported again.
As far as the previous test runs have been considered, most of the errors that have been caught are related to broken urls or some other small problems that are usually fixed with a single commit so I don't think that a single report would be re-sent that often.
If this functionality is required then I'll be more than happy to add it. We can add some sort of a log/cache type structure that checks if a given report with similar content has been sent to the mailing list in a given time interval, let's say 15 days for now.
Regards,
Akshat Jaimini
On Wed, Oct 4, 2023 at 12:08 AM Magnus Hagander <magnus@hagander.net> wrote:
On Tue, Oct 3, 2023 at 8:24 AM Akshat Jaimini <destrex271@gmail.com> wrote:
>
> Hello everyone,
> The testing harness has been successfull at catching multiple issues in pgweb in the past few months. The reports are for now being sent to me via email in case of any failure.
>
> Is it ok if I can setup the harness to send the error reports directly to the mailing list?
Hi!
I think that would be reasonable, *if* it is ensured that a report is
only sent once. That is, if it finds the same issue on a later run, it
must not re-send the same thing. How does it work in regards to that
today?
--
Magnus Hagander
Me: https://www.hagander.net/
Work: https://www.redpill-linpro.com/
Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list.
От
Daniel Gustafsson
Дата:
> On 3 Oct 2023, at 21:30, Akshat Jaimini <destrex271@gmail.com> wrote: > > That is, if it finds the same issue on a later run, it must not re-send the same thing. How does it work in regards tothat today? > > As per the current flow whenever a new commit is pushed to the pgweb repo, the tests are executed. If some tests fail,an error report is sent with the information of all the failed tests. So if that particular issue has been resolved,the same report won't be sent but if some other commit is pushed without resolving that particular issue then thatparticular error will be reported again. That doesn't seem terribly great, while bugs and errors should be fixed when found, sending reports of them repeatedly risk reporting-fatigue. Publishing this report to a website would handle that I think. One question, would this test harness detect and report potential security issues like XSS? If so we should probably limit the audience of the report.. -- Daniel Gustafsson
> Publishing this report to a website would handle that I think.
I had sent a proposal/tried to start a discussion for this a few days earlier : https://www.postgresql.org/message-id/CAMaW3Vg%2BGoQ3JPNo%2BfbLk9ajQv%3D4g4J-bzSAH0OJL7S71_qMig%40mail.gmail.com
It would actually make the reporting mechanism a lot easier if we can publish the results to a website. I am currently working on a small prototype in golang. Currently the reports are being stored as artifacts on Github actions(only available for 90 days) but we can use services like Supabase etc to store our reports and present them on the website. Once we integrate supabase we can get rid of Github artifacts for good.
> One question, would this test harness detect and report potential security issues like XSS?
Security related tests were not added in the Gsoc timeline but we are planning to add them. Maybe when we add those tests we can create a separate section on the proposed website only available to some 'admins' with all these sensitive reports being displayed there.
We can actually benefit with some more discussion on this.
Regards,
Akshat Jaimini
On Thu, Oct 5, 2023 at 8:32 PM Daniel Gustafsson <daniel@yesql.se> wrote:
> On 3 Oct 2023, at 21:30, Akshat Jaimini <destrex271@gmail.com> wrote:
> > That is, if it finds the same issue on a later run, it must not re-send the same thing. How does it work in regards to that today?
>
> As per the current flow whenever a new commit is pushed to the pgweb repo, the tests are executed. If some tests fail, an error report is sent with the information of all the failed tests. So if that particular issue has been resolved, the same report won't be sent but if some other commit is pushed without resolving that particular issue then that particular error will be reported again.
That doesn't seem terribly great, while bugs and errors should be fixed when
found, sending reports of them repeatedly risk reporting-fatigue. Publishing
this report to a website would handle that I think.
One question, would this test harness detect and report potential security
issues like XSS? If so we should probably limit the audience of the report..
--
Daniel Gustafsson
Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list.
От
Daniel Gustafsson
Дата:
> On 6 Oct 2023, at 08:05, Akshat Jaimini <destrex271@gmail.com> wrote: > > > Publishing this report to a website would handle that I think. > I had sent a proposal/tried to start a discussion for this a few days earlier It would probably help if you could link to a report from a run of the test suite. I clicked through the linked repo but I was unable to see an example testrun. > > One question, would this test harness detect and report potential security issues like XSS? > Security related tests were not added in the Gsoc timeline but we are planning to add them. Maybe when we add those testswe can create a separate section on the proposed website only available to some 'admins' with all these sensitive reportsbeing displayed there. For tests like that we must really think about scope, limiting the report isn't useful if we publish the tests for anyone to run themselves and thus generate the report. Malicious actors are no doubt probing the website continuously regardless of this, but we don't necessarily need to do the job for them. -- Daniel Gustafsson
> I clicked through the linked repo but I was unable to see an example testrun.
You can find the reports here: https://github.com/destrex271/pgweb-testing-harness/actions/runs/6189299124 . You can check the 'report', 'test-log' and 'failure_logs' artifacts, the other ones are experimental for now.
> For tests like that we must really think about scope, limiting the report isn't useful if we publish the tests for anyone to run themselves and thus generate the report.
> Malicious actors are no doubt probing the website continuously regardless of this, but we don't necessarily need to do the job for them.
Oh yes, that is a valid point, I guess we might need to separate these tests then in some private repo? I don't know if this is possible though but we can think of some other approaches. Because if we keep those tests publicly available that will just create more problems for us, as you mentioned in your reply.
I'll try to find more approaches to this because the private repository does not seem to go with the idea of open source. I might be wrong about this, so please let me know if I am wrong.
Regards,
Akshat Jaimini
On Fri, Oct 6, 2023 at 6:09 PM Daniel Gustafsson <daniel@yesql.se> wrote:
> On 6 Oct 2023, at 08:05, Akshat Jaimini <destrex271@gmail.com> wrote:
>
> > Publishing this report to a website would handle that I think.
> I had sent a proposal/tried to start a discussion for this a few days earlier
It would probably help if you could link to a report from a run of the test
suite. I clicked through the linked repo but I was unable to see an example
testrun.
> > One question, would this test harness detect and report potential security issues like XSS?
> Security related tests were not added in the Gsoc timeline but we are planning to add them. Maybe when we add those tests we can create a separate section on the proposed website only available to some 'admins' with all these sensitive reports being displayed there.
For tests like that we must really think about scope, limiting the report isn't
useful if we publish the tests for anyone to run themselves and thus generate
the report. Malicious actors are no doubt probing the website continuously
regardless of this, but we don't necessarily need to do the job for them.
--
Daniel Gustafsson
Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list.
От
Daniel Gustafsson
Дата:
> On 6 Oct 2023, at 19:12, Akshat Jaimini <destrex271@gmail.com> wrote: > > You can find the reports here: https://github.com/destrex271/pgweb-testing-harness/actions/runs/6189299124 <https://github.com/destrex271/pgweb-testing-harness/actions/runs/6189299124>. You can check the 'report', 'test-log' and'failure_logs' artifacts, the other ones are experimental for now. Thanks, that was a bit hidden (which is a Github UI issue and not something against this work). > I'll try to find more approaches to this because the private repository does not seem to go with the idea of open source.I might be wrong about this, so please let me know if I am wrong. Just because a project is open source doesn't mean that everything about it needs to be done in public. Security teams and security processes generally operate behind closed doors, to avoid leaking vulnerabilities before they can be patched, and then publish their work and findings once there is a remedy (either as an advisory with a CVE or some other form). -- Daniel Gustafsson
> Security teams and security processes generally operate behind closed doors, to avoid leaking vulnerabilities before they can be patched, and then publish their work and findings once there is a remedy.
Ok! So we can then proceed with a private repository maybe? We can fork the CI setup from the current testing harness and just add the respective security tests. The generated report can then be accessed by the security team/any concerned individuals in the deployment team. I'd be happy to host this repo if needed for now.
> Thanks, that was a bit hidden
Yup this is one of my main concerns with only relying on github actions also there are multiple runs for the monitoring cron job as well so these test runs usually get lost in the list. As a temporary solution I had added the github action run url in the email being sent and the reports attached with that email.
I have started working on the website to view these reports, will be sharing the development prototype url shortly.
Regards,
Akshat Jaimini
On Mon, Oct 9, 2023 at 6:12 PM Daniel Gustafsson <daniel@yesql.se> wrote:
> On 6 Oct 2023, at 19:12, Akshat Jaimini <destrex271@gmail.com> wrote:
>
> You can find the reports here: https://github.com/destrex271/pgweb-testing-harness/actions/runs/6189299124 <https://github.com/destrex271/pgweb-testing-harness/actions/runs/6189299124> . You can check the 'report', 'test-log' and 'failure_logs' artifacts, the other ones are experimental for now.
Thanks, that was a bit hidden (which is a Github UI issue and not something
against this work).
> I'll try to find more approaches to this because the private repository does not seem to go with the idea of open source. I might be wrong about this, so please let me know if I am wrong.
Just because a project is open source doesn't mean that everything about it
needs to be done in public. Security teams and security processes generally
operate behind closed doors, to avoid leaking vulnerabilities before they can
be patched, and then publish their work and findings once there is a remedy
(either as an advisory with a CVE or some other form).
--
Daniel Gustafsson
Hey everyone,
Sorry for the delay(got stuck in some uni stuff). I am working on this right now, so currently I am planning to use Supabase to store the failure report content instead of storing entire files. This will also give us a way to prevent the harness from reporting any repeating errors in subsequent builds and make it easier to display the data on the website for the harness(which can then be hosted easily on Cloudflare pages or whatever is preferred by the community). Does this seem to be in the right direction or should we opt for some other solution?
Regards,
Akshat Jaimini