ui element selection #246643
ui element selection #246643
Conversation
src/vs/workbench/services/host/electron-sandbox/nativeHostService.ts
Outdated
Show resolved
Hide resolved
src/vs/workbench/contrib/chat/browser/chatEditing/simpleBrowserEditorOverlay.ts
Outdated
Show resolved
Hide resolved
|
|
||
| const buf = captured?.toJPEG(95); | ||
| return buf && VSBuffer.wrap(buf); | ||
| } | ||
|
|
||
| async getElementData(windowId: number | undefined, offsetX: number = 0, offsetY: number = 0, token?: CancellationToken): Promise<IElementData | undefined> { |
There was a problem hiding this comment.
What if this takes a URI of the webview to target instead. To start picking, you would read the current URI (or maybe origin?) of the active (or last-active) webview from the IWebviewService and then find the target matching that URI here. Then you would not be tied to requiring the simple browser.
There was a problem hiding this comment.
yeah that's a good idea. i think will maybe defer to next iteration and try this out
|
|
||
| const margin = model.margin; | ||
| const x = margin[0]; | ||
| const y = margin[1] + 32.4 + 35; // 32.4 is height of the title bar, 35 is height of the tab bar |
There was a problem hiding this comment.
This probably varies per platform and whether the custom titlebar is enabled or not
There was a problem hiding this comment.
i'm gonna try to pivot over to screenshotting the whole window + highlighting the element. if i can get that working, we won't need any funky calculations to grab screenshots like this
There was a problem hiding this comment.
Not sure what kind of model quality you get, my maybe outdated knowledge of vision is that they quantize subsections of the image into tokens and so a larger screenshot means the selected element gets less 'attention'. Though maybe adding some large buffer like 100px on each side around the selected element is good enough to give some extra context and titlebar buffer without going to the entire screen.
There was a problem hiding this comment.
yeahhh, i just tried both scenarios and it is very wonky how the screenshotting wants to work. the screenshot grabs everything on top, so having the buffer might capture things outside of the frame.
currently looking into how we can properly get the x/y/width of the frame, but i think i'll leave for now and explore other options next iteration, not sure if there is another good way to do this.
src/vs/workbench/contrib/chat/browser/chatEditing/simpleBrowserEditorOverlay.ts
Outdated
Show resolved
Hide resolved
src/vs/workbench/contrib/chat/browser/chatEditing/simpleBrowserEditorOverlay.ts
Outdated
Show resolved
Hide resolved
src/vs/workbench/contrib/chat/browser/chatEditing/simpleBrowserEditorOverlay.ts
Outdated
Show resolved
Hide resolved
src/vs/workbench/contrib/chat/browser/chatEditing/simpleBrowserEditorOverlay.ts
Outdated
Show resolved
Hide resolved
| for (const group of groups) { | ||
|
|
||
| if (!(group instanceof EditorGroupView)) { | ||
| // TODO@jrieken better with https://github.com/microsoft/vscode/tree/ben/layout-group-container |
There was a problem hiding this comment.
🙈 dupes the todo I should have taken care of ;-). @bpasero maybe it's time we make this more "offical"
| } | ||
| } | ||
|
|
||
| export class SimpleBrowserOverlay implements IWorkbenchContribution { |
There was a problem hiding this comment.
Not sure where we are going this but if editor overlays become "a thing" then we should make this more proper. We should have some kind of service where you register for an editor type and it calls you back to populate your overlay
src/vs/workbench/services/host/electron-sandbox/nativeHostService.ts
Outdated
Show resolved
Hide resolved
|
If the changes appear safe, you can manually trigger the pipeline by commenting |
|
I filed #247499 for follow up debt. |
allow users to attach UI elements to chat